Besides there were the prisoners in Castles Thunder and Lightning.

The essential features of the plan were that at a preconcerted signal we at the second and third floors should appear at the windows with bricks and irons from the tobacco presses, which a should shower down on the guards and drive them away, while the men of the first floor would pour out, chase the guards into the board house in the basement, seize their arms, drive those away from around Libby and the other prisons, release the officers, organize into regiments and brigades, seize the armory, set fire to the public buildings and retreat from the City, by the south side of the James, where there was but a scanty force of Rebels, and more could be prevented from coming over by burning the bridges behind us.
It was a magnificent scheme, and might have been carried out, but there was no one in the building who was generally believed to have the qualities of a leader.
But while it was being debated a few of the hot heads on the lower floor undertook to precipitate the crisis.

They seized what they thought was a favorable opportunity, overpowered the guard who stood at the foot of the stairs, and poured into the street.

The other guards fell back and opened fire on them; other troops hastened up, and soon drove them back into the building, after killing ten or fifteen.

We of the second and third floors did not anticipate the break at that time, and were taken as much by surprise as were the Rebels.