When he came back to England, and found himself avoided by everybody, the Moonstone was thought to be at the bottom of it again.

The mystery of the Colonel's life got in the Colonel's way, and outlawed him, as you may say, among his own people.
The men wouldn't let him into their clubs; the women--more than one--whom he wanted to marry, refused him; friends and relations got too near-sighted to see him in the street.
Some men in this mess would have tried to set themselves right with the world.

But to give in, even when he was wrong, and had all society against him, was not the way of the Honourable John.

He had kept the Diamond, in flat defiance of assassination, in India.

He kept the Diamond, in flat defiance of public opinion, in England.