I'm open for inspiration :-)
What would you consider an "actual castle type structure"? A huge square donjon? A "palas"-style representational hall? Or something else altogether?
The attached photograph shows a couple of modules I designed assembled into a solitary tall round donjon. I could probably redesign it into a square one with a more "Norman" look fairly easily.
In case you haven't seen it yet, here's my collection of Modular Castle Playset compatible things:
Tall walls are a good idea, but a bit tricky if everything is to be nicely modular. I'll have to think about this!