The river crossing at Packhorse Ford, located about a mile downstream from Shepherdstown, could not meet the needs of a growing population. Thomas Van Swearingen began operating a ferry in 1765. This location provided easy access to towns on both sides of the river including Charles Town and Harpers Ferry, VA and Hagerstown, Frederick, and Baltimore, MD. In 1775 Van Swearingen had constructed a "Ferry Inn" at the landing on the Maryland side of the river. The community that grew as a result of the ferry became known as Bridgeport.
John Blackford acquired interest in the ferry and adjoining acres through an inheritance from the Van Swearingen family when he married Sara van Swearingen in in 1805. Sadly, she died at the age of 25. He remarried Elizabeth Knode in 1812 and it was then that he built Ferry Hill. Looking for an ideal location to build a home and start his family he decided to build the house high on the bluff overlooking the Potomac River. The land was fertile, and the nearby river, with a convenient ferry crossing, would facilitate delivery of his crops to market. Eventually the farm grew to over 700 acres.
“Seeing a beautiful mansion perched on the summit of a lofty eminence, on the opposite shore, … crossed the river. After a pretty fatiguing walk up a moderate mount, I found myself on a level plain, where sits the mansion, or palace, rather of---. It is built in the form of an L, and is the most splendid building of any country house in the state, and view from it is equally grand.” –Anne Royall, 1828
In 1833 Blackford sold 41 acres, 3 rods, 1 perch (5 1/2 yards) to the Chesapeake and Ohio Canal Company. John Blackford continued to operate the ferry and utilized the canal to transport goods.
As transportation needs grew, the ferry proved inadequate. Franklin Blackford sold the ferry and surrounding property to the Virginia and Maryland Bridge Company, and by 1850 a toll bridge was constructed.