The Doctor's been travelling on his own for decades, if he's telling the truth about his age. And we saw what happened to Ten when he was alone for months—not nearly as long as Eleven—and Eleven is just full of so much self-loathing that I cannot stand it…
Ten's character relationships were almost more…transient. He travelled with Rose, and then he lost her but found Martha, and then lost her and found Donna again. He was a person who needed interaction, needed human contact, and got it almost constantly—except after he was forced to wipe Donna's memories. Eleven, on the other hand…
Since almost the moment he regenerated, he's had someone there, but they're so few in comparison. He met seven-year-old Amelia Pond, crash-landed in her backyard, and then popped away for a five-minute trip and came back to find her nineteen. He was selfish and stole her away and she travelled with him, and everything was wonderful... And eventually, they even pulled Rory along for the ride!
Amy's funny and vivacious and always up for an adventure, and even if Rory's a little more reserved, Eleven loves him anyway. He loves poking fun at Rory Williams, but all he can think is that he's one of his best mates, and how perfect he is for Amelia Pond: the girl who waited, the girl who will forever be the one with the crack in her wall and the hope in her eyes, just like she was so many years ago.
He doesn't have nearly as many friends in this incarnation as he did in his last; there's Amy and Rory and River who are there at first, relative constants in his life, and then there are others like Craig who he cares for just as much but only sees upon occasion. He's lost contact with Martha and Wilf and Jack and Sarah Jane and he knows it's probably for the best, because they grew to care for and love his last incarnation—the one with the mop of hair and the manic, expressive eyes and the pinstriped suit and the multicolored converse. He's the same person, but he's not, and he knows they won't understand—knows it won't be the same—so he never goes back.
And he convinces himself that it's okay like this because isn't it really? Amy and Rory are wonderful and kind and perfect and he can't ask for better friends... But as time goes on and terrible things happen, he realizes that their story is going to end the same as Rose's and Martha's and Donna's. And he can't stand to think that this will happen, because even if his previous incarnation loved so expansively he loves maybe more intensely, and he can't think of what he will do if Amelia and Rory are ruined beyond repair.
(It would be entirely his fault.)
So he buys them a cozy little house with a door the color of the TARDIS, drops them off like they've shared a cab and yet there's just so much more between them. He knows it's for the best but he just can't stand to see them go, because they've been there since the start of him and he's not sure what he's going to do without them.
But he knows his own happiness is nothing compared to theirs, so he does his best, and plasters a smile on his face, and straightens his bow tie, and sets out on his own into the vast universe with only his TARDIS and his memories at his side.
And when he travels, when he visits a planet with a sky the color of Amy's hair or meets someone who reminds him so much of Rory in his loyalty and his compassion, he feels that bitter pang of regret and selfishness. I should go back and get them, he thinks, because surely they miss me just as much as I miss them and maybe they'll drop everything, drop their whole lives just to travel with me like Rose and Martha and Donna did. But then he realizes that this will never happen, because they're married now and they have lives and friends and jobs, and if they have all of that how can they possibly make time for their Raggedy Doctor? He's just a figment of their imagination, a nice memory to think about over tea. What is a silly little alien who's lived for far too long when compared to a normal, human life, after all?
So he keeps his distance while he can, and when he visits Earth he smiles sadly at the advertisements emblazoned boldly with "for the girl who's tired of waiting," because he knows it's true and he's expected far too much from Amelia Pond. He knows it's his own damn fault he's this lonely and he should just get on with his life...
But then the Daleks abduct the three of them, and he can see the distance they're standing apart and it's too far too far too far because the Ponds have always been attached at the hip. Rory still can't believe she said yes, and Amy loves him so much it hurts, but what's wrong this isn't right and he knows it's all his fault.
(After all, if he hadn't meddled in their lives, if he had never pushed himself into Amy's world when she was so young and impressionable, they would surely be happy and married with children by now. They'd be raising a Melody Pond who looks just like her father with her mother's eyes, and even if that means he would never have met River either, it doesn't matter because at least they would be happy.)
But this didn't happen, will never happen, and he hates himself for it, but there's nothing he can do about it now so he does what he thinks is right. It's all he's ever done: he works to fix these lives he's broken beyond repair.
(Even if he doubts he's ever truly helped anyone in his life.)
And when they ask him to bring them home, again and again and again because we need to get our marriage back on track and that was enough excitement for now—dinosaurs on a spaceship? really?—and our friends will start asking why we're aging faster than we should be—he agrees without question, hides his tears and his disappointment behind his ever-cheery facade. After all, he's just the Raggedy Doctor to them; he realizes that's all he's ever been; and no matter how much he loves Amy and Rory Pond, they will never love him that much in return because he simply loves too much.
There's nothing he can do about it; maybe it's a result of the abstract idea that love originates in the heart, and he's got two of those so maybe this is just another curse of the Time Lords. He doesn't know, and maybe he doesn't care, because if he doesn't have the Ponds he has nothing, and what is he supposed to do now? So he treasures the time he has left with them, travels and deduces and realizes that soon, he's never going to see them again. (It's breaking his hearts.) But he also knows that it's his fault anyway, and this is all he deserves, so he smiles like a child and wears fezzes and stetsons and makes cracks about horses and dry cleaning because that's all he has left.
He hates himself, knows that everyone he's ever touched has been ruined because of his own selfish desire, but he can't seem to make himself stop. He is nothing without his friends, but he is nothing to his friends; it is an endless paradox of agony that will haunt him forever.
But he can't let on about this weakness, because that would shake their strangely immovable idea that he's this untouchable demigod. For some reason, even if they love him so much less, they seem to think him brilliant and infallible. And no matter how wrong they are, he has to keep pretending.
(Because if he doesn't, he thinks he might finally destroy himself.)