First of all, the speaker of "Skunk Hour" is stuck on an island. We find this out right away. So his human interaction is limited to who's stuck there with him and, judging by the way he describes his neighbors in the first part of the poem, those people aren't his best friends. Though the speaker seems to know a lot about what these folks are like from the outside, he doesn't seem to know the more intimate details of their lives. This leads us to draw the conclusion that, besides seeing them on a regular basis, he doesn't actually know them very well. When night rolls around, this becomes even more evident as he drives around looking for signs of human life, and finds nothing but skunks.
The speaker prefers to be alone because he doesn't want to face ridicule for his mental illness.
He's not actually alone. This poem was just a brooding, lonely moment. He really has tons of friends on the island.