On his return he had an important interview with Henry Clay, of whose political party he had for several years been the acknowledged leader in Alabama.

He urged Clay to place himself at the head of the movement in Kentucky for gradual emancipation.

Upon Clay's refusal their political cooperation terminated.

Birney never again supported Clay for office and regarded him as in a large measure responsible for the pro-slavery reaction in Kentucky.
Birney, who had now become discouraged regarding the prospect of emancipation, during the winter of 1831 and 1832 decided to remove his family to Jacksonville, Illinois.

He was deterred from carrying out his plan, however, by his unexpected appointment as agent of the colonization society in the Southwest--a mission which he undertook from a sense of duty.
In his travels throughout the region assigned to him, Birney became aware of the aggressive designs of the planters of the Gulf States to secure new slave territories in the Southwest.