I don't think that the term "crazy" quite suits someone like the Doctor. Enigmatic? Complicated? Completely bananas? Those seem more suited to him.
I think that it's less the muse and more the fandom itself that drives me crazy. How complicated is it to write a pup with no canon? Very. He's Loomed, he's got a grandkid, he's half-human, he's more than a Timelord, and, oh yeah, there are many websites that are dedicated to estimating how old he really is.
Let's see, what do we know? We know that Peter Moffat once said that Doctor Who has no canon. KThanxdood, srsly. 40+ years of time and space travel does tend to contradict many things, and the Doctor has bounced around in age, time traveling, and even his species' genetic makeup.
And then there's the Time War. How bloody little is known about the Time War except that it killed all the Gallifreyans and a bunch of races, and that the Doctor was on the front lines. Speaking of not knowing, what's the Doctor's name? We don't know! Not that I think I'd want canon to suddenly supply me with one, but, still. It's the frustration of not knowing when you write for a character.
In the end, I'm forced to create my own "canon", deciding that the Doctor was Loomed, yes, with the Other and a human woman (mentioned only in the TVM), but into the form of a child, not an adult, as Lungburrow implies (and is contradicted in Black Orchid and Smith and Jones, with references to the Doctor's youth). He is sterile, but being Loomed from the Other, he inherited Susan as a granddaughter, as she was his biological granddaughter, and the imprint from the Other on the Doctor made her recognize him (canon, Lungburrow).
But then, canon goes around and contradicts itself. Again.
"I was a dad, once." - 2.11, "Fear Her"
Soooooo, we get a rewrite. The Doctor is sterile, but, with another Timelord, Loomed a child for them. The child grew up, Loomed its own child, Susan.
These constant rewrites tend to drive me crazy.