We don't get much information about where our speaker comes from or about the nearby village in this poem, but we do know that he's far away from civilization. We also know that the man who owns the woods lives in town in a house. From this little information, we can deduce that if you own things (like the owner of the woods does), then you live in the midst of society. Our speaker is not so concerned with society. In fact, society to him is about as appetizing as cod liver oil. He'd rather be alone with nature. To us, the village sounds quaint, cute, and warm. To our speaker, the village represents his obligations, responsibilities, and promises.
Society is all about ownership and owning things in the world of this poem.
The village represents obligation, debt, and promises made.