In the view of most white people, it was the white man who was emancipated.

The white districts had no desire to let the power return to the Black Belt by giving the Negro the ballot, for the vote of the Negroes, they believed, would be controlled by their former masters.
Johnson's adoption of Lincoln's plan gave notice to all that the radicals had failed to control him.

He and they had little in common; they wished to uproot a civilization, while he wished to punish individuals; they were not troubled by constitutional scruples, while he was the strictest of State Rights Democrats; they thought principally of the Negro and his potentialities, while Johnson was thinking of the emancipated white man.

It is possible that Lincoln might have succeeded, but for Johnson the task proved too great..