Rooney, so nicknamed to distinguish him for his cousin and contemporary Fitzhugh Lee of “Clermont,” Fairfax County, was the Lees' second son, born in 1837. He was one of the liveliest and most likable of the Lee children. He was adventurous and as a child evoked his father's praise couched in jest. Lee referred to him as “too large to be a man, too small to be horse” and believed he needed a tight rein. When he was eight years old, Rooney cut off his the tips of the forefinger and middle finger on his left hand while playing with a set of straw cutters.
Rooney's adventures during the 1850s kept him away from Arlington much of the time. He entered Harvard in 1854, one of the three Virginians at the school. At Harvard, he was popular and quickly fell in with Boston society. He demonstrated his athletic prowess, pulling an oar on the Harvard crew. He did not remain at Harvard to graduate, however.
In 1857, with the aid of General Winfield Scott, he secured a commission and fought in the campaign of 1858 against the Mormons. When the fighting was over, however, he became bored and by 1859 had given up the army and married Charlotte Wickham. Rooney and Charlotte settled down to farm the White House, the estate on the Pamunkey River in New Kent County, Virginia he had inherited from his grandfather, George Washington Parke Custis.