The earliest portion of this two-part house is a traditional colonial Virginia frame dwelling built in 1764 by Dr. Thomas Walker, a colonial leader and explorer of the west. Here in 1781 Walker's wife delayed the British colonel Banastre Tarleton to give the patriot Jack Jouett time to warn Thomas Jefferson and the Virginia legislators of Tarleton's plan to capture them. The stately brick portion, an example of Jeffersonian classicism by master builder John M. Perry, was erected in 1823-24 for William Cabell Rives, minister to France, U.S. Senator, and Confederate congressman. Columned conservatories were added in 1844 by William B. Phillips. Rives's granddaughter Amelie, wife of the Russian painter Prince Pierre Troubetzkoy, was a novelist and playwright. She and her husband made Castle Hill their home in the early decades of the 20th century. Castle Hill is noted for its extensive gardens and landscaped grounds.
Castle Hill is located .8 mile northwest of Rte. 231, 2 miles NE of its intersection with Rte. 600 in the Cismont vicinity. It is a private residence and is not open to the public.