The fourth side was open to the spectators in the hall, and behind the ropes at the back there sat in the centre of the row of chairs a fat red-faced man in evening-dress who was greeted on all sides as Colonel Joe.

"Colonel Joe" was the referee, and a person on these occasions of great importance.
There were several preliminary contests and before each one Colonel Joe came to the front and introduced the combatants with a short history of their achievements.

A Hindu boy was matched against a white one, a couple of wrestlers came next, and then two English sailors, with more spirit than skill, had a set-to which warmed the audience into enthusiasm and ended amid shouts, whistles, shrill cat-calls, and thunders of applause.
Meanwhile the heat grew more and more intense, the faces shinier, the air more and more smoke-laden and heavy.
Shere Ali came on to the stage while the sailors were at work.

He exchanged a nod with "Colonel Joe," and took his seat in the front row of chairs behind the ropes.
It was a rough gathering on the whole, though there were some men in evening-dress besides Colonel Joe, and of these two sat beside Shere Ali.
They were talking together, and Shere Ali at the first paid no heed to them.

The trainers, the backers, the pugilists themselves were the men who had become his associates in Calcutta.