The river crossing at Packhorse Ford, located about a mile downstream from Shepherdstown, could not meet the needs of a growing population. Thomas Van Swearingen began operating a ferry in 1765. This location provided easy access to towns on both sides of the river including Charles Town and Harpers Ferry, VA and Hagerstown, Frederick and Baltimore, MD. In 1775 Van Swearingen had constructed a "Ferry Inn" at the landing on the Maryland side of the river. The community that grew as a result of the ferry became known as Bridgeport.

John Blackford acquired interest in the ferry and adjoining acres through an inheritance from the Van Swearingen family when he married Sara van Swearingen in 1812. Looking for an ideal location to build a home and start his family he decided to build the house high on the bluff overlooking the Potomac River. The land was fertile, and the nearby river, with a convenient ferry crossing, would facilitate delivery of his crops to market. Eventually the farm would grow to over 700 acres.

In 1833 Blackford sold 41 acres, 3 rods, 1 perch (5 1/2 yards) to the Chesapeake and Ohio Canal Company. John Blackford continued to operate the ferry, however, now the canal would also be utilized to transport goods.

As transportation needs grew the ferry was proving inadequate. Franklin Blackford sold the ferry and surrounding property to the Virginia and Maryland Bridge Company and by 1850 a toll bridge was constructed.

It was its location that drew attention to Ferry Hill Place again in 1861. Henry Kyd Douglas lived there with his parents, the Reverend Robert Douglas and his wife Helena. When war broke out Henry enlisted in the Army of the Confederacy. The Federal Army looked upon the family with mistrust. The family was held under house arrest for most of the war. They were instructed to keep the shutters closed. One stormy evening a shutter was blown open. The Union Officers saw this as an act of treason, implying the Reverend was signaling to the Confederates across the river. Reverend Douglas was arrested as a spy. Although he was never formally charged, he was held at Fort McHenry for several months before being allowed to go home.

The property passed on to Nannie Cowen, a daughter of John and Helena, who with her husband ran a pig farm from 1914 through 1928. Times were hard but the Beckenbaughs continued to struggle on. They opened a restaurant in 1948. Even after they sold the property it remained a restaurant until 1974

The link with John Blackford was severed in 1951 when the house was sold to Frederick Morrison. It provided a perfect location for a restaurant. Many students from Shepherd College recall enjoying an evening of dining and dancing at Ferry Hill. It was during this period that extensive changes were made to the house. The imposing columns facing the river were added. The wall separating the kitchen from the dining room, and the servant's staircase were removed. An addition was added to the back of the main house and many of the out buildings were torn down.

Because of its location along the Chesapeake and Ohio Canal, it served as the Headquarters of the Chesapeake and Ohio Canal National Historical Park from 1979 until 2001.

Historic Ferry Hill Place still stands in an idyllic location proudly overlooking the Potomac River, waiting for the next stage of its life to begin.