"[...] but, as a rule, I like to be alone and undisturbed." (2.13)
Just to make sure we understand, the Invisible Man comes right out and tells us that he prefers to be alone.
Communication with the world beyond the village he had none. (4.2)
In case we thought that the Invisible Man had some particular beef with Iping, we learn pretty quickly that he actually doesn't talk to anyone. (The opening of Chapter 4 is actually all about how the Invisible Man is isolated from the community.)
"I was alone…In all my great moments I have been alone." (19.37)
At the beginning of his talk with Kemp, the Invisible Man makes a connection between being alone and being awesome. We'll see if he keeps this opinion for the rest of the book. (Spoiler alert: he doesn't. Check out the last quote in this theme.)