A/N 1: This scene in Season 2 where Mr. Carson tells Mrs. Hughes that he's decided to leave Downton to work at Haxby is one of my favorites out of all 3 seasons. So I had to write a story about it. I hope you enjoy it.

A/N 2: This story started out as a one-shot, but has grown in my mind as a continuing story (you can blame Mrs Dickens 713 for it). This chapter is the original one-shot, the only differences being this author's note and the fact that this chapter ends "To be continued" rather than "The End."

Mrs. Hughes had known when she'd found Mr. Carson reflecting on the quality of the cellar at Downton that he'd made his mind up. She'd asked for confirmation, but he hadn't had to really say the words, for which he was thankful. Deciding to leave Downton had been one of the hardest decisions he'd ever made. In time he would be able to think and speak of it without pain, but that time had not yet come.

"And just when we thought we were getting back to normal," she mused.

"Don't tell me you'll miss me," he said, in what he hoped was a lighthearted tone. The atmosphere felt heavy, and he hoped to lighten it by giving her an opening to tease him a little.

She didn't speak immediately, her eyes darting up to his and away again a few times before she finally held his gaze. "I will, Mr. Carson. Very much. And it costs me nothing to say it," she said softly.

He almost couldn't speak, he was so surprised. He was quite touched and after a moment he forced himself to respond, unwilling to let her kind words go unacknowledged. "Thank you," he said at last, trying to smile, though he felt rather gloomy. "That means a lot to me."

And there they stood looking at each other, both still and silent, until a loud noise from down the corridor broke the tension, and they left the room to make their way to the kitchen. They walked together, he a little behind, and neither of them spoke. He thought she seemed uneasy. He could see only part of her face, so he studied the set of her shoulders and the rhythm of her steps, but they told him nothing. He could never read her as well as she could him. Later, at dinner, he could see her better, but although she was preoccupied, she didn't appear to be ill or upset.

He invited Mrs. Hughes to have some sherry with him late that evening, but she turned him down, saying she was tired. She certainly looked tired, though he would never tell her so, and she seemed distracted, with that same uneasy expression he'd noticed before dinner.

He watched her back as she walked away from him, down the dim and deserted passage. She walked standing just as firm and straight as always, but her step was slower. He could tell, because though he didn't usually watch her when she left his pantry, he always listened. He could hear her heels clicking on the floor, the consistency of her pace always comforting to him. She turned her head slightly to one side and he knew she must have heard him, somehow caught him watching her without even seeing him, because she immediately quickened her pace and within seconds she was out of sight.

He wondered if her mind was troubled by thoughts of his leaving. Perhaps she would not be leaving her home as he would, but she would have to make adjustments with a new butler coming in, to establish a rhythm with someone else. After the chaos of the war, she would now have to manage the chaos of a new man coming in as head of household staff. It would take time and patience on her part. He had thought about this in regard to Haxby as well. He would be working with a new housekeeper, and new staff would be hired. It would be like starting with a blank slate, really, which could be both good and bad. Lady Mary would be in charge of hiring the housekeeper, of course, but she would accept, perhaps even request, his input.

He walked back into his pantry and poured himself a glass of sherry. Sitting down heavily, he wondered what it be like working without Mrs. Hughes. He worked with another housekeeper several months out of every year when the family went to London for the Season and at other times when the Crawleys opened Grantham House. He and Mrs. Winters got on well and she was perfectly competent, but she was no Mrs. Hughes. There were little things she didn't do quite as efficiently as Mrs. Hughes. She also didn't share his taste for reading novels as Mrs. Hughes did, so they had less to talk about when they sat down for a cup of tea together from time to time. And there were countless other things about her that made little difference individually, but added up to his preference of one housekeeper over the other. He enjoyed London, but he couldn't think of a more welcome sight than her smiling face at Downton when he returned with the Crawleys. Several years ago, almost on a whim, he'd returned a day before the family. He did it on the pretense that he wanted to get the heavy cases back and unpacked before the family arrived, but he couldn't have explained the real reason, except that he had felt restless and impatient to return to Yorkshire. There was as much to do as ever once he'd gotten to Downton, but he and Mrs. Hughes had had an extra day to catch up, before the Crawleys once again descended on the Abbey. Ever since then, he'd made it his routine every year to return from London a day early. He had a quieter homecoming when he returned separately from the family and the other staff, and he preferred it that way. He went in through the back door without any fuss, and if he was lucky she was in her sitting room when he arrived. If not, he went and found her. She was a refreshing sight, always with a smile and a brisk welcome, and they fell back into their routine almost instantly, as though he'd never been away at all.

It might be possible to find another housekeeper as efficient as Mrs. Hughes and it might be possible to establish such a smooth working relationship with her, but what seemed less likely to him was that this other housekeeper would also become as good a friend to him as she was. He shook his head. In truth, he wouldn't want that. He didn't want to try to mold some poor woman into a second Mrs. Hughes. That would be a project doomed to failure from the beginning. No, he was moving to Haxby Park, not New York, and he would still see Mrs. Hughes occasionally. He had no desire to replace her.

Mr. Carson poured himself a second glass of sherry. It would be impossible to replace her, he knew, just as it would be impossible for Haxby Park ever to be as much of a home to him as Downton Abbey was. He would not have her there, her tidy little figure swaying as she walked up and down the corridors, checking every bedroom, those ever-present keys jingling at her hip. She would not be there to scold him when he overworked himself or to take care of him when he was sick. If she would, he thought it might almost seem like home to him. He considered bringing this suggestion to Lady Mary for a fraction of a second, but dismissed the idea almost immediately. He regretted leaving Downton in need of a butler; he would not deprive the Crawleys of their housekeeper as well. But more than that, he knew Mrs. Hughes would never accept even if Lady Mary were to make such an offer. She was not devoted to Lady Mary as Mr. Carson was, and she did not like Sir Richard Carlisle at all. And even though her history there was not as long as his, Downton was still her home.

Mr. Carson set down his now-empty glass and frowned. It was past time for him to go to bed. He locked his pantry and headed up to the attics. His thoughts were so troubled he feared he wouldn't sleep very well tonight. As he slowly undressed, his thoughts returned to Mrs. Hughes. He wondered how often he might be able to see her once he went to Haxby. At Downton he was entitled to a half-day every fortnight, but he did not often take it. There always seemed to be so much to do. Additionally, he did not have any family outside of Downton, so there was nothing to draw him away, though Lord Grantham would have gladly allowed him to take time to visit friends if he had wanted to. Perhaps he should try to change that habit when he went to Haxby. His family was at Downton Abbey, and he would want to visit. Perhaps Mrs. Hughes would visit him as well, come have tea with him on her half-day. There would be no more evening chats over sherry or leftover wine, though, and he would miss them. They discussed the running of the house when they sat together like this, but they also shared tidbits of gossip, talked about books they had read, told stories, and simply enjoyed each other's company. In the evening she relaxed, her eyes softened, and she smiled more. She looked so lovely in the mellow light of evening and her voice charmed him. He found it fascinating that she seemed to have different ways of speaking for different situations, and he wondered if she knew that her Scottish accent was much more subdued when she spoke to the family than it was when she was downstairs, and that it grew much thicker when she was angry - speaking sternly to her maids, refusing to hand her store cupboard key over to Mrs. Patmore, and quarrelling with him. He loved to hear her speak, but her voice in the evening was the one he loved best. It was sweet and low in his ears, its lilting, skipping cadence singing a little melody like none he'd ever heard. And no matter which accent she was using, he loved the way she said his name.

As he climbed into bed and pulled the blanket over his body, it occurred to Mr. Carson that his deciding whether to go to Haxby or to stay at Downton came down to choosing between the two women he loved, Lady Mary Crawley and Elsie Hughes. Somehow the realization that he loved Mrs. Hughes did not shock him. He laid in his bed, staring at the ceiling, and felt only wonder. He'd given up thoughts of love and having his own family once he'd advanced beyond a certain level in service. He had known at some point that he would be a single man for the rest of his life. And yet here he was in love, at his age. He was not an old man, but he'd thought himself past the age of falling in love. Not because love was restricted to the young, but because the routine of his life didn't include the sort of activities he considered instrumental to falling in love: going to dances and fairs and socializing outside of Downton Abbey. He spent an odd hour here and there reading a novel at a local teashop, or enjoying some of the entertainments of the Metropolis, but for the most part Charles Carson worked and slept. He'd got it wrong by doing his socializing while working and then falling in love with a colleague. Well, he hadn't got anything wrong, exactly, for he hadn't been trying not to fall in love. He simply hadn't expected it. Perhaps he should have. Perhaps he should have known the moment Elsie Hughes had walked in the back door of Downton Abbey for the first time that his heart would eventually belong to her. He had taken over as butler some years before her arrival as head housemaid, so he had made his decision years before that to curtail his personal life in favor of his work. She was an intelligent and attractive woman, but even if he had not sworn off romance himself, it became clear within a few months that Mrs. Hughes, still Elsie then, had similar ambitions. It was perhaps because of this similarity of purpose that they struck up an easy friendship. She was just the sort of servant Mr. Carson always approved of; she was not afraid of hard work and discipline, knew how and when to hold her tongue, and always seemed to be more aware of what was going on than anyone around her. She was a bit younger than he was, but they were of the same generation and had some interests in common, so they had naturally gravitated toward each other. The housekeeper had hired Elsie with a view toward training her own replacement, and although Mrs. Trent had stayed on another several years before handing over her keys, in that time Elsie proved herself as reliable and capable as her references had said she was. Mr. Carson had been well-satisfied to see her take her place beside him at the servants' dining table and in the sitting room down the corridor, knowing that she would be a great professional ally and friend.

And she was more valuable even than he had expected. Their leadership styles were complementary and they fit together so well that sometimes he wasn't sure where he ended and she began. She took care of her side of running the house, but she also watched out for him, very subtly at first, but more overtly as time passed. She wasn't always successful in her attempts to keep him well. He had stubbornly resisted her very insistent demands that he stop working so hard, and been rewarded by being confined to bed for several days after a very undignified collapse in the dining room.

As he had told Lady Mary when she first asked him to come to Haxby, it would be a wrench to leave Downton. It was not only because of Mrs. Hughes that this was so, but leaving her would be the hardest thing to bear. Still, he felt that Lady Mary needed him, and he would not fail her. It was never a matter of choosing which woman he loved best. That was something he could never do. That would be like deciding whether one loved one's wife or one's daughter more. The love for a wife and the love for a daughter were so very different that it seemed wrong even to compare the two. No, it was a matter of who needed him most. Mrs. Hughes could take care of herself and would carry on without him, but Lady Mary needed him, now probably more than ever. That's what he had to think of, rather than the two houses. Haxby would never compare to Downton in any way. The house was grand, but there would be nothing and no one in it, save Lady Mary, that compared to the things and people at Downton that would make it so hard to leave. He was sorry to have to make such a painful choice, without ever having had the joy of having his own wife or daughter. He had loved Lady Mary since she was a child, but his heart whispered that if he were trying to choose the one he loved best, he would not be leaving Downton. However, he dismissed that as an unprofitable line of thought. He had made his decision, had told Lady Mary after dinner. What could he offer Mrs. Hughes, after all? And would she even want it, or him? He knew she cared about him, but he had no idea what her deeper feelings might be. No, he would leave Downton. He would regret it every minute of every day, but he would go.

To be continued...

Please review if you can spare a moment.