We must believe this, since for some reason or other he did come out from his retreat for a while. Perhaps it was only to see whether there were any letters for him at the Tesmans.
I don't know.
No one knows.
But this reappearance shows that his detachment from the world was not complete.
And incompleteness of any sort leads to trouble.