It is true that the clearances are very tight. I have done this intentionally because of the small tolerances. I used an 8mm drill to widen the hole of the handle a little. If this does not work, you can consider sanding the base part down a bit but this was not needed for me.
I think that the risk of the holder not being able to hold properly over time is too big when the clearance gets too large. However, I have not tested this since the method of drilling it out afterwards works for me but I will take a look into this. Thanks for your comment, this will help further development of this holder.