With the completion of the dock, the mainland’s extensive hardwood forest was harvested. Kelderhouse continued buying land and began to process cordwood for sale to passing ships by building a sawmill near what is now the John Burfiend farm. Over the next 30 years, Port Oneida grew to include a blacksmith shop, a boarding house, general store and post office, two barns, and the Kelderhouse residence. Kelderhouse owned most of these buildings as well as nearly half of the land on Pyramid Point. In 1866, he bought a gristmill on the CrystalRiver from John Fisher.
Lumbering drastically altered the appearance of the landscape. By the 1890’s, most of the land had been logged off and most Great Lakes steamships were burning coal. Unable to compete with larger operations such as that of D.H. Day in Glen Haven, the dock and mill were sold. The loss of this industry and the death of Thomas Kelderhouse in 1884 led to the demise of the Kelderhouse fortune.
By 1908, all the buildings at the original Port Oneida town site, except the Kelderhouse residence, had been abandoned. The Kelderhouse family lived in this house until 1934, when it was sold to Fred Baker. In 1944, the boarding house was torn down, and by 1952 the other buildings and apple orchard were removed. The wood was used in constructing the Barratt barn and the Burfiend pig barn.