One thing lead to another and this thing is completely rebuild & a new concept builds upon the existing servo arm, resulting in less fiddling with tolerances and a bigger, more rigid part (as the printed adapters are weaker).
While the squeeze-fit solutions and such where nice, I decided on adding a M2 nut to the M5 threaded rod holder.
Thanks to FlouSH, Talpa and also Zippy for the ideas & comments on the initial design(s).
Print standing on the hole, worked best for me.
Laying can be done too, but the servo arm holder is weaker (as the small parts may break away).
On the UP I used the smallest support settings but more dense perimeters/infill.
Note: The M5 adapter could be tighter, but otherwise the delivered servo screws won't fit in.
Assembly: See pictures.
PS: The UP software seems to crash sometimes when exporting the STL from Sketchup and importing it, even though netfabb did not find any holes. Running it through the fix-routine and saving it again (even though nothing happened) will solve this.