The river crossing at Packhorse Ford, located about a mile downstream from Shepherdstown, could not meet the needs of a growing population. Thomas Van Swearingen began operating a ferry in 1765. This location provided easy access to towns on both sides of the river including Charles Town and Harpers Ferry, VA and Hagerstown, Frederick and Baltimore, MD. In 1775 Van Swearingen had constructed a "Ferry Inn" at the landing on the Maryland side of the river. The community that grew as a result of the ferry became known as Bridgeport.
John Blackford acquired interest in the ferry and adjoining acres through an inheritance from the Van Swearingen family when he married Sara van Swearingen in in 1805. Sadly she died at the age of 25. He remarried Elizabeth Knode in 1812 and it was then that he built Ferry Hill. Looking for an ideal location to build a home and start his family he decided to build the house high on the bluff overlooking the Potomac River. The land was fertile, and the nearby river, with a convenient ferry crossing, would facilitate delivery of his crops to market. Eventually the farm would grow to over 700 acres.
In 1833 Blackford sold 41 acres, 3 rods, 1 perch (5 1/2 yards) to the Chesapeake and Ohio Canal Company. John Blackford continued to operate the ferry, however, now the canal would also be utilized to transport goods.
As transportation needs grew the ferry was proving inadequate. Franklin Blackford sold the ferry and surrounding property to the Virginia and Maryland Bridge Company and by 1850 a toll bridge was constructed.