The Negro was regarded as "the ward of the nation," but he was told impressively that "labor is a public duty and idleness and vagrancy a crime." All wanted him to work: the Treasury wanted cotton and other crops to sell; the lessees and speculators wanted to make fortunes by his labor; and the army wanted to be free from the burden of the idle blacks.

In spite of all these ministrations, the Negroes suffered much from harsh treatment, neglect, and unsanitary conditions.
During 1863 and 1864, several influences were urging the establishment of a national bureau or department to take charge of matters relating to the African race.

Some wished to establish on the borders of the South a paid labor system, which might later be extended over the entire region, to get more slaves out of the Confederacy into this free labor territory, and to prevent immigration of Negroes into the North, which, after the Emancipation Proclamation, was apprehensive of this danger.
Others wished to relieve the army and the treasury officials of the burden of caring for the blacks and to protect the latter from the "northern harpies and bloodhounds" who had fastened upon them the lessee system.
The discussion lasted for two years.

The Freedmen's Inquiry Commission, after a survey of the field in 1863, recommended a consolidation of all efforts under an organization which should perpetuate the best features of the old system.

But there was much opposition to this plan in Congress.