Located in back of the main house are two rectangular buildings, which are set at right angles to the house, forming a small service court. These buildings, the two surviving slave quarters which housed slaves who were the house servants of the Custis and Lee family, have three rooms each, and have stone foundations with rough stucco walls featuring Greek Revival architectural details. It is thought that Hadfield also planned these buildings. The stone well is located between one of these structures and the North Wing of the house.

The Summer Kitchen was located in the North Slave Quarters and housed the carriage driver, Daniel and his son, Daniel in one room. George Clark, the long time plantation cook, and his assistant lived in another room. The “Summer Kitchen” was located in a basement of this building, but was filled in at some point and no longer exists.

The South Slave Quarters housed Selina Gray, Mrs. Custis's personal maid and trusted housekeeper. She, her husband and their eight children lived in one room with a small loft where some of the children slept. The loft was accessible by ladder and the crawl-space attic had a ceiling only high enough for small children. There were no windows in the attic.

The middle room in the South Quarters building was used as a Smoke House where hams and other meats would be hung from the ceiling to smoke and cure. The third room in this building housed other slaves that worked in the Custis-Lee household.

There was a slave School House located in the grove of trees behind the flower garden and roughly where the Old Amphitheatre of the National Cemetery is now located. Slave field hands lived in log cabins, mostly in the southern end of the plantation, but none of these cabins have survived.