Agnes or "Wig" was the third of the four Lee daughters and the fifth of seven children. She was born in 1841. Agnes spent much of her time in reading, studying, playing piano and in working in her garden. She did little housework beyond taking care of some of her own things or straightening up Annie's. Agnes kept a fascinating journal during her childhood years, later published and entitled Growing Up in the 1850s.
Before going away to boarding school in 1855, she and Annie had a tutor, Miss Sue Poor, from whom they learned music, English composition, French, and probably arithmetic. For a time she helped to instruct the Arlington slaves by conducting a Sunday evening school for them and by instructing individual children before and after breakfast.
Agnes took a lively interest in the changes made in the house especially the refurnishing of the large hall in 1855. She loved Arlington dearly, but admitted that, compared to West Point, it was hardly clean and neat. She was religious and was confirmed in the Episcopal Church in 1857.
Agnes was a charming and attractive young lady, and there is some evidence that she felt a romantic attachment to Orton A. Williams, her mother's young cousin and a frequent visitor at Arlington, just before the Civil War. Her father is said to have frowned upon the romance because he regarded young Williams as too unsettled to marry.
People in Lexington after the war found her somewhat reserved and aloof. In part this may have been due to the tragic death of Orton Williams in 1862, and to her own serious illness in 1865, which according to her father, left her steady and regular but without "velocity." Considered her mother's favorite daughter, Agnes never married and died from typhoid fever in October 1873, at the age of 32.