The U.S. Army Corps of Engineers began work on Fort Point in 1853. Due to the relatively short range of artillery at the time, plans specified that the lowest tier of artillery be as close as possible to water level so cannonballs could ricochet across the water's surface to hit enemy ships. To accomplish this, the ninety-foot cliff at the construction site was blasted down to a mere fifteen feet above sea level. The new structure had three tiers of casemates (vaulted rooms housing cannon), seven-foot thick walls, and a barbette tier on the roof with additional guns and a sod covering to absorb the impact of enemy cannon fire. While more than thirty such forts existed on the east coast, Fort Point was the only one of its kind on the Pacific.
With the 1861 outbreak of the Civil War, the Army mounted the first fifty-five guns at Fort Point. By October 1861, there were sixty-nine guns in and around the fort, consisting of 24, 32, and 42-pounders, as well as 8 and 10-inch Columbiads. During the Civil War, as many as five hundred men from the 3rd U.S. Artillery, the 9th U.S. Infantry, and the 8th California Volunteer Infantry were garrisoned at Fort Point. Stationed several thousand miles from the major theaters of combat, the men spent their days in a routine of drills, artillery practice, inspections, sentry duty, and maintenance chores. Enlisted men bunked twenty-four to a casemate on the third tier; officers had single or double quarters on the floor below. To supplement coal fuel, soldiers gathered driftwood from the shore for fuel. Though prepared for attack, Fort Point never fired its guns in defense during the Civil War.