At Camp Nelson, soldiers were housed in various tents and barracks across camp and, toward the end of the camp's existence, in a multitude of buildings, including the hospital wards, the soldiers' home, and the recruiting rendezvous. About 40 percent of Kentucky's USCT passed through Camp Nelson by the end of 1865. Kentucky's USCT fought in a number of battles and skirmishes in Virginia, Tennessee and Kentucky and performed garrison duty at numerous strategic points in Kentucky.
Camp Nelson eventually became a refugee camp for the soldiers' wives and children. Soldiers brought their families with them when they enlisted at the camp. Their protection and aid at Camp Nelson was debated until December 1864 when the Army established the "Home for Colored Refugees" in the southwestern corner of the camp. Teachers and missionaries were sent by the American Missionary Association (AMA) to educate the refugees, and doctors and nurses were provided by the U.S. Army. The refugee camp eventually contained 97 duplex cottages in three rows, a school, mess hall, hospital, reception ward, commissary, store rooms, barracks, laundry, lime kiln, offices, surgeons' and teachers' lodgings, and a private sutler's establishment (a follower of an army camp who peddled provisions to the soldiers). The refugee camp was built to house, feed and educate the families of the USCT and at some points housed more than 3,000 people.
Following the June 1866 closure of Camp Nelson, the refugee camp school and other buildings were purchased by the Freedman's Bureau and administered by the AMA. The cottages continued to be lived in by the same families. The longest term missionary to the camp, Rev. John G. Fee later bought 130 acres, including the refugee camp, and sold or leased lots back to the residents. This area became the community of Ariel, now known as Hall.
Almost all of Camp Nelson's buildings were dismantled and taken away in 1866 or soon after. Many of these camp building were intended to be temporary structures, and the only standing building from the Civil War era is the Oliver Perry House or "White House," which housed the Quartermaster and Commissary offices, and was recently renovated by the Jessamine County Fiscal Court. The area still retains rich archeological evidence, in addition to several earthen forts and other original landscape features that convey an appearance similar to the Civil War encampment. The Camp Nelson National Cemetery and a cemetery established for the refugee camp also remain.