Yes, the speaker of "Skunk Hour" knows the people of Nautilus Island. Yes, he lives there year-round while others come for summer then leave for the rest of the year. But while his permanent address is on Nautilus Island, and he's got a back porch to stare out from, we get the sense that he doesn't feel quite at home in this town. Simply the way he describes the place puts it at such a distance that it's hard to imagine he feels much comfort, or much fondness, for it at all. What makes a home is hard to define, but we think it's safe to say that home is not where you lay your head every night, but – yup, you guessed it – where the heart is.
The speaker is just like the other vacationers. He doesn't live there year-round, but has come back for one day during the off-season.
The speaker used to feel at home there, but now that the mystery person has left, he's feeling too lonely to feel homey.