The Marine Hospital was also one of the most controversial buildings of Charleston's antebellum period, viewed by state's rights advocates as an illustration of the Federal government's abuse of its powers. The Marine Hospital Fund, established in 1798 by an Act of Congress, was one of the earliest manifestations of Federal involvement in the area of public health, and provided the first Federal health care and social welfare initiative for United States sailors. Funding for the hospitals was deducted from sailors' wages and supplemented by shipping industry taxes. In Charleston, many people resented the heavy hand of the Federal government in the construction of their Marine Hospital, which began in 1831. Even though Mills had left the city only two years earlier, state's rights supporters were particularly infuriated by the replacement of their local architect and contractors with Mills and other professionals from Washington D.C., as well as the increased costs of the project. By the time of its completion in 1834, the Marine Hospital was rejected by Charlestonians as an unworthy civic accomplishment.
After the Civil War, the Episcopal Church used the building as a free school for black children. From 1895-1939 the structure served as the Jenkins Orphanage, whose famous band traveled the United States and Europe on fund-raising tours. In 1939 the Housing Authority of Charleston purchased the building for administrative offices, a purpose it still serves today.