The town of Lewes, originally called Swanendael, was first settled in 1631 by a group of colonist from the Netherlands. However, they where quickly annihilated by the Siconese, a local Native American Tribe. In 1659, a former area of Swanendael was refortified by the Dutch fostering the settlement of Lewes. At this time it was called Hoerenkill by the Dutch and later Whorekill by the English. Over the next few year the town would be raided and burned by Lord Baltimore's Solider's in near by Maryland, and then the town would be taken over by the English from the Treaty of Westminster.
The Ryves Holt House was built in 1665, it is one of the few building that managed to survive Lord Baltimore’s Raid that occurred in 1673. Shortly after the raid the Dutch regained authority of the area for a short time before a treaty was established granting the land to the English. After William Penn’s arrival to New Castle in 1682 he renamed the lower three counties of Pennsylvania (now Delaware) New Castle, Kent, and Sussex and gives Lewes its namesake that exists today. Eventually, Ryves Holt House was purchased in 1723 by its namesake, Ryves Holt, who served as the first Chief Justice of Delaware from 1745 until his death in 1763. This house is thought to be one of the oldest homes in Delaware.