The Invisible Man appeared to be regarding Kemp. "Because I've a particular objection to being caught by my fellow-men," he said slowly […]
"Fool that I am!" said the Invisible Man, striking the table smartly. "I've put the idea into your head." (17.97, 99)
When the Invisible Man calls the people who would catch him "my fellow-men," he seems to recognize some common quality between him and the people around him – even those he worries will betray him.
"It occurred to me that the radiators, if they fell into the hands of some acute well-educated person, would give me away too much, and watching my opportunity, I came into the room and tilted one of the little dynamos off its fellow on which it was standing, and smashed both apparatus." (20.52)
The Invisible Man worries that his equipment will betray him. That is, he worries that these radiators will tell his secret. There's some irony here because Griffin is telling this story to Kemp, who <em>will </em>tell Griffin's secrets. Notice also that Griffin's solution here is violence, which will also be his solution when it comes to Kemp. We're seeing a trend here.
"So last January, with the beginning of a snowstorm in the air about me—and if it settled on me it would betray me!—weary, cold, painful, inexpressibly wretched, and still but half convinced of my invisible quality, I began this new life to which I am committed. I had no refuge, no appliances, no human being in the world in whom I could confide." (22.1)
Again, betrayal is connected to isolation. Griffin can't confide in any human being, and now, even the snow could betray him. Seem a little – or a lot – paranoid? Absolutely. That's probably why the movie versions of Griffin tend to show him as clinically insane.