I believe he was so full of her aspects that he did not want to look in her direction when there were other people to hamper his imagination.
You understand I am piecing here bits of disconnected statements.

Next day Flora saw him leaning over the field-gate.

When she told me this, I didn't of course ask her how it was she was there.

Probably she could not have told me how it was she was there.

The difficulty here is to keep steadily in view the then conditions of her existence, a combination of dreariness and horror.
That hermit-like but not exactly misanthropic sailor was leaning over the gate moodily.