Robert E. Lee's pacing could be heard from downstairs. He had retired to his bedroom where the internal struggle between loyalty to state or nation would be decided. Eventually the pacing ceased. When the footsteps resumed Lee was descending the stairs, holding two letters. He handed the letters to his wife saying, “Well Mary, the question is settled. Here is my letter of resignation, and a letter I have written to General Scott.”
Robert E. Lee's resignation from the U.S. army in the early morning hours of April 20, 1861 would bring to an end to the thirty year relationship between Lee and Arlington House. Within two days he would board a train bound for Richmond, never to return. His wife did not accompany him and remained at Arlington House to await developments reluctant to abandon her family's ancestral home. She would not remain for long, however.
Having spent most of his army career in the Corps of Engineers, Lee recognized that Arlington House's geographical prominence made Mrs. Lee's continuing residence problematic. With the house and the surrounding land perched upon dominating ground, overlooking the United States Capital it would not be long before United States forces would attempt to occupy Arlington Heights.
Within five days of departing for Richmond Lee wrote his wife, “War is inevitable, and there is no telling when it will burst around you….You have to move and make arrangements to go to some point of safety which you must select.” As the weeks progressed and Union forces began concentrating in Washington it became more apparent that Mrs. Lee would have to prepare her House for evacuation.
Mrs. Lee had no intention of permanently abandoning the House and was slow at first to act upon her husband's advice. In a letter to her daughter Mildred on May 5, she stated, “I would not stir from this house, even if the whole northern army were to surround it.” On May 11, she referred to her husbands worries. “Were it not that I would not add one feather to his load of cares,” she would have remained at Arlington come what may.
In the end, however, it was Robert E. Lee's concerns and a report of immediate occupation that drove her to evacuate. The warning shot across the bow came in early May in the form of Orton Williams, a cousin who came to Arlington House with alarming news. He warned Mrs. Lee that she must pack up her valuables immediately as the Union army was preparing to cross the Potomac and occupy Arlington in a matter of hours.
Though Orton's warning turned out to be a false alarm, it was an event that crystallized the need to prepare the house for an orderly evacuation. Mrs. Lee's arthritis restricted her role in the physical work of preparing the house for evacuation but she supervised the hasty preparations. The family silver was packed away along with the personal papers of George Washington, her father George Washington Parke Custis and her husband Robert E. Lee and were sent to Richmond.
Mrs. Lee informed her husband of the evacuation's progress on May 9, 1861. “I suppose ere this, dear Robert, you have heard of the arrival of our valuables in Richmond. We have sent many others to Ravensworth & all our wine & stores, pictures, piano etc.”
Six days later, on May 15, 1861, Mrs. Lee entrusted the keys to the house to Selina Gray, the chief house slave, and the Lees left Arlington. The family headed first to the White House Plantation and eventually on to Richmond. Selina and the rest of the Arlingtons slaves were to run the estate as though the Lees were to any day return.
On May 23, 1861 Virginians voted 96,750 yay, 32,134 nay, in favor of secession. The next day, the United States Army crossed the Potomac and occupied the defensible ground around Washington, including the Arlington Estate. Arlington House itself was not immediately occupied by Union forces.