He had an older sister named Sarah who died giving birth to her first child on January 20, 1828. He had a younger brother named Thomas Jr. who died as a child about 1814.