Over the years, Lee's military career kept him away from Arlington much of the time. However, the house and its occupants were very important to him and he regarded the estate as his home. In the 1850s he began to play a more important role in the affairs at Arlington, taking over some financial matters for his father in law, George Washington Parke Custis. After Custis' death in 1857, Lee took leave from the army to act as executor of the Custis estate and manager of Arlington and the other Custis lands. Though Arlington House never belonged to Lee (it belonged to his wife), he was the effective master of Arlington after 1857. Between 1857-1861, he attempted to reorganize the slaves into a more efficient labor force, cleaned up the grounds, hired a new overseer and supervised the planting of crops. He also oversaw extensive rebuilding around the plantation. He virtually rebuilt the overseer's house at the farm and the stable west of the mansion. He also fixed the roof of the mansion and took out a fire insurance policy on the mansion and the barn.
Lee and his wife took different attitudes toward a number of matters, and neatness was one of them. Lee was disciplined, punctual, precise, and a very careful dresser who was fond of social functions. He liked parties, the theater, and the company of women. He was courtly and genial with equals and courteous and hospitable to strangers. He preferred things to be orderly and clean. In all these habits and tastes he differed from his wife. But they were a devoted and congenial couple.