Although he is usually associated with Mount Vernon on the Potomac River, George Washington spent much of his youth at Ferry Farm on the Rappahannock River, several hundred yards downstream from Chatham. He often returned to Fredericksburg to visit his family and friends. On many occasions George Washington visited Chatham to see his good friend William Fitzhugh. On April 29, 1786, Washington recorded in his diary that he had "set out from the Bowling Green a little after sunrise. Breakfasted at General Spotswood's. Dined at my Sister Lewis' in Fredericksburg and spent the evening at Mr. Fitzhugh's of Chatham." During Washington's visits, he enjoyed the lavish hospitality and entertainment for which Fitzhugh was well noted.
There is also evidence that during these visits, William Fitzhugh and George Washington conferred about the new and improved agricultural practices of the day. In 1786, Fitzhugh sent some seeds from the huge crabapple to Washington with the promise that the "fruit of the seedling crab is larger, more juicy and supposed to make a good yellow cyder [sic] and in greater quantity than fruit." Fitzhugh and Washington served on the vestry of Pohick Church together. Fitzhugh was one of the last people to see Washington alive.
The marriage of Fitzhugh's daughter, Mary Lee Fitzhugh, to George Washington Parke Custis, the adopted grandson of George Washington, reveals the close relationship between the Fitzhugh and Washington families. Mary Ann Randolph Custis, the daughter of Mary Lee and George Washington Custis, married Robert E. Lee.
Today a portrait of George Washington hangs at Chatham to serve as a reminder of the warm relationship between Washington and Fitzhugh. The original painting dates from the early 19th century. The artist is unknown.