Since i have not found any good driver for 4mm bits i made one myself. Done in OpenSCAD, source provided.
Inside is an hole for a magnet (4mm diameter, 2mmheight). Magnet should just hold by pressfiting, otherwise use glue.
Tolerances work well for Prusa MK2/MK3, but maybe you have to adjust.
Edit: you seem to like this one, so i made it parametric (including customizer support). You can now generate hex drivers for 2mm up to 10mm - including different magnet-sizes.