Born on the plantation of John Walker near Columbus, Georgia, Haralson was raised as a slave and was self-educated. He was sold on the auction block in Columbus to J.W. Thompson. When Thompson died he became the property of J. Haralson of Selma. He remained Haralson's slave until 1865. While a slave, he was self educated and identified as a preacher.

​

In 1874 he was elected to the United States House of Representatives. He maintained a friendship with the former president of the Confederacy, Jefferson Davis, which earned him criticism from his Republican associates.

​

Haralson's oratorical abilities drew the commendation of Frederick Douglass.