In September 1861 Smith mustered into service as a lieutenant colonel of the 54th Ohio Infantry. The following month, he was made colonel of this same unit. He distinguished himself during the Vicksburg campaign under Grant. In August 1863, he was promoted to brigadier general and commanded a brigade of the XV corps. He served in the Red River campaign. After the failure in this campaign, he served in the pursuit of the Confederate cavalry under General Nathan B. Forrest. He resigned his position due to health problems in 1864. He was brevetted a major general in March 1865.
