Maybe adding .1 or .2 mm to the outer diameter isn't a bad idea.
After using them for a bit, I found that the dust is probably becoming quite hot. The additional two images show my PLA print, after using the 20mm drill with only 22mm inner diameter. So be careful to leave enough space for the dust to be removed, don't overdo it and consider using e.g. ABS for this one.