Judging from the statements of Julia Strong and the butler in the Whitmore house, it is obvious that Whitmore sent a letter to Mrs.Collins, with whom he was in love.

Something transpired to make him regret having sent the note and he decided to steal it out of the post office.

He was caught before he had succeeded in 'jimmying' the door, so that the letter must have been delivered at the Collins house.
I take it, from the threats which Collins made against Whitmore, that he intercepted the note and that a lively scene between him and his wife followed.
"As for Whitmore, he did a most sensible thing.

He kept his identity effectually concealed.

Before arriving at the post office he had disguised himself in cheap, shabby clothes, so that when he was captured no one thought he was other than an ordinary burglar.