James Henry Gooding was born into slavery on August 28, 1838 in North Carolina. At a very young age his freedom was purchased by a James M. Gooding, who may have been his father, and he was sent to New York City. On September 11, 1846, Gooding was enrolled as a student in the New York Colored Orphan's Asylum, a prominent school and boarding house run primarily by Quaker women. There he received a classical education and became a proficient and prolific writer, a talent which would serve him for the rest of his life. From 1850-1852 he was indentured out of the Asylum to work for an Albert Westlake. As he approached adulthood he made the decision to hide his past as a slave, and began telling people he was born free in Troy, New York. In 1856, at the age of 18, he took a job on board a whaling ship out of New Bedford, Massachusetts. Whaling was one of the few industries at that time in which an African-American man could find employment on equal footing with whites. He often composed poetry describing life at sea on whaling ships. During his voyages he made as much as $20 per month, a salary equivalent to an officer on board ship. He gave up whaling in the fall of 1862, when he settled down in New Bedford and married Ellen Allen in the Seaman's Bethel, a prominent church in the city.
He did not remain at home in New Bedford very long. Six days before Gooding's marriage, President Abraham Lincoln announced the Emancipation Proclamation, which took effect January 1, 1863 and opened the door for the enlistment of African Americans into the Union armies. James Gooding enlisted on February 14, 1863 into what would become the 54th Massachusetts Infantry. Throughout his service he sent letters to editors of the New Bedford Mercury, which published them weekly and tracked the progress of the 54th Massachusetts.
The 54th Massachusetts was sent to the barrier islands of coastal Georgia and South Carolina. Providing a description of the South to the Mercury, Gooding remarked in June 1863 that all he saw was "…stink weed, sand, rattlesnakes, and alligators. To tell the honest truth, our boys out on picket look sharper for snakes than they do for rebels." This monotony would not last long. Less than a month later, Gooding participated in the storming of Fort Wagner near Charleston, SC, depicted in the 1989 film Glory. Describing the action at Fort Wagner to the Mercury, Gooding said, "We met the foe on the parapet of Wagner with the bayonet…"