He went because he felt that he must go somewhere.

After the conversation with Asaph, he simply could not remain at home.

If Phoebe Dawes called, he knew that he must see her, and if he saw her, what should he say to her?
He could not tell her that she must not visit the Cy Whittaker place again.

If he did, she would insist upon the reason.
If he told her of the "town talk," he felt sure, knowing her, that she would indignantly refuse to heed the malicious gossip.

And he was firmly resolved not to permit her to compromise her life and her future by friendship with a social outcast like himself.