Following a series of treaties and further conflicts with the American Indians in the area, the white population of Abbeville County grew steadily. In the decades following 1800, the cultivation of cotton and the importation of slaves to produce it steadily grew in Abbeville County.  From 1790 to 1800, the slave population in the County increased from 18.1 to 21.9 percent. By 1820, slaves accounted for 41.5 percent of Abbeville County's entire population.

James Edward Calhoun was a member of one of the most prominent pioneer white families in the South Carolina Piedmont. Local tradition holds that Calhoun joined the United States Navy while still in his teens.  He traveled the world and learned seventeen languages. By all accounts, Calhoun was a brilliant man whose interest ranged from ethnography  to machinery to astronomy. Calhoun owned Midway Plantation and would later develop the Millwood Plantation.

Members of the Calhoun extended family, such as his uncle Andrew and brother John, initially governed the operation of the Midway Plantation. Throughout the period of James Edward's absentee ownership, the slaves of Midway Plantation produced a consistent cotton crop, with sixty-five bales produced in 1823, fifty bales in 1826 and sixty bales in 1827.  However, one year later, in 1828, John C. Calhoun told James Edward that the return from staple crops hardly compensated the expense of cultivation because the prices of land and slaves were at "depressed rates."

The economic problems faced by James Edward Calhoun were compounded by problems presented by his manager Speed. Speed, a manager for seven years on the plantation, started trouble when William Clark, a new manager, came on the scene.  A letter to Clark mentions a slave rebellion, possibly instigated by Speed, that occurred either at Midway or at an adjoining Plantation.

By early 1831, James Edward was thinking about giving up the sea for the plantation. In a letter to a relative, John Calhoun, he mentioned that he would not want to live at Midway again but would rather prefer building on one of his other tracts. This was his first mention of what would be later known as the Millwood Plantation. He resigned from the Navy in late 1833 or early 1834 and began to concentrate fully on planting.

Millwood Plantation was home to James Edward Calhoun's slaves. Millwood had been in operation for 33 years when slavery ended in 1865.

In an effort to expand his enterprise in the spring of 1832, Calhoun began what would become his major operation, Millwood Plantation, by sending " two men, one woman and one girl" there to form a settlement. He also hired a man named Absalom Roberts at $100 a year to oversee and to labor alongside his slaves there. At this time, sixty acres were cleared and planted in corn and peas. On July 30th, James Edward "began a Crib Dam at the upper part of Trotter's Shoals, Savannah River, intending to erect Mills at [his] Millwood place, the site where Trotter had his built before the Revolutionary War."

With these activities, Millwood Plantation was born. In November 1832, Calhoun bought a 422.5 acre piece of land adjoining Millwood, called the Hamilton Tract, for $4 an acre, or $1,690, payable in three annual installments. The following month he brought another 200 acres for $400 from Richard Lovington. In addition, Calhoun purchased two islands in the Savannah River, containing 4.5 acres.  These real estate acquisitions demonstrate Calhoun's serious intent to build a large planting empire and to pursue the planter's life in earnest. In the 1830's, Calhoun had an estate that included three different plantations: the Hamilton Tract, which he called "Stockdale," Millwood, and Midway, where he made his home. He had an overseer and a slave work force at each place.

Although 61.8 percent of the 55 slaves owned by Calhoun in 1830 were between the ages of 10 and 55, Calhoun was nonetheless rich in land and poor in labor. His work force was enlarged by purchases and through natural increase. Almost nothing is known about Calhoun's slaves as individuals and little is known about how they lived or how they were treated.

Some evidence about Calhoun's early slave force is provided in a unique source, an unsigned slave ledger. The ledger consists of three parts. The first section contains a list of names and numbers that probably represent the ages of the slaves Calhoun owned in 1834. Some of the names are preceded by a "x" or have a line drawn through them, suggesting that they were either sold or dead. A second section of the ledger presents a list of names and dates from February 1835 to June 1847. These are probably slave births. The third section of the ledger provides a list of slaves who, in 1839, comprised an as yet unidentified agricultural work force called the Edgefield Gang.

One hundred and twenty-six slaves are listed in the first section of the ledger. Among males the highest mortality rate was among those slaves who were younger than ten years old.  Among females, the highest mortality rate occurred in the ten to twenty-four year old class.  The information provided in the second part of the ledger also provides limited evidence that Calhoun was an active slave buyer.  Although seventy-two slaves were enumerated in 1834 and 155 in 1840, the ledger lists the birth of only 49 children of which 43 or 72.9% survived.  Regardless of slave fertility and mortality rates, the available evidence indicates that a sizable percentage of Calhoun's slaves were within the optimal working ages.

In 1830, 61.8% of his slaves were between ten and fifty-five.  In 1840, 54.2% were and in 1850, 63.4% of his slaves were with in this age group. Ten years later, 66.2% of his slaves were between the ages of ten and fifty-five.  In 1840, 89 of Calhoun's slaves were employed in agriculture, and at least three were involved in manufacturing and trades.  Coupled with Calhoun's urge to mold his three plantations into one major planting concern was his desire to develop a strong mechanical component within his empire.  Millwood, located at Trotter's Shoal in an area already known for its mills, was well suited to such activity.

At about the same time Calhoun began using his slaves to make Millwood a well integrated agricultural and mechanical enterprise, he was beginning to think about moving his residence there. In March 1834, he wrote in his diary: "At Millwood getting our stuff for an overseer's house but which I shall occupy until I have leisure to build there a better house." At this point, James Edward Calhoun became, at age 36, the resident planter of Millwood Plantation.  In the 1830's he sought to expand his interest even further by asking his brother-in-law, John C. Calhoun, to inquire into certain operations for him.

James Edward was perhaps contemplating expanding the manufacturing side of Millwood.  The gold deposits often found in the South Carolina Piedmont and frequently corresponding with John C. Calhoun about exploring his lands had intrigued James Edward. During the same period, it seems that he was contemplating the construction of a cotton-bag factory at Millwood. In 1843, he sent to New York for a loom for weaving cotton bagging, Osnaburg [coarse cotton fabric] and Negro clothing.

Two years later, he corresponded with a factory superintendent in Tallahassee, Florida. This machinist assured Calhoun that he could have his $2,000 worth of machinery in operation before too long. Another factor that played on James Edward's mind in the early 1840's was the insolvency of his brother John Ewing Calhoun.

James Edward was deeply affected by the failure of his brother and he was probably all too well aware of his own possible failure and how he had tried to stretch his finances to the limit. On February 4, 1839, 41-year-old James Edward married 23-year-old Maria Edgeworth Simkins. Maria did not live on the Millwood Plantation right away due to its rustic condition. James Edward and Maria communicated mainly by mail because of the fifty-mile distance apart. In 1844, Maria became seriously ill when dangerous epidemics ravaged the South Carolina Piedmont. She did not recover from her illness and died sometime before June 29th. Throughout this period, James Edward continued to use his slaves to build his plantation, although he again tried to sell Millwood in 1848 for $300,000.

The census for 1850 shows that Calhoun used two hands at his water powered mills to process corn, wheat, and logs. In addition, the Millwood slaves produced seventy bales of cotton, a variety of other crops, and used livestock worth $1,869. The 1860 census indicates that Calhoun's plantation continued to prosper and that his slaves produced sixty-three bales of cotton, a thousand pounds of tobacco, and many other crops, including $1000 worth of orchard products.  The value of the livestock on the plantation rose to $9,608.

When the first shots of the Civil War were fired in Charleston Harbor in the spring of 1861, James Edward Calhoun was sixty-three years old. He assisted in the war effort by sending slaves to Charleston to work.  In spite of his opinions about the political situation that faced South Carolina planters, James Edward Calhoun continued to develop Millwood. Millwood Plantation had been in operation for thirty-three years when slavery ended in 1865.