Where It All Goes Down
Although the setting is hardly described at all, it plays a major role in this poem. The poem opens "In a field" and that's all we get. No description of the field, no landscape descriptions, no animals, no people, no weather, no time of day etc…just, "field." So if it's so important, why doesn't he describe it to us?
What's important for our speaker is to create a contrast between himself and his presence with the setting of the poem. It isn't the particulars of the field that are important, but rather, how the speaker exists within relation to the field. So, while he's in a field, he takes up some space, which he refers to as "the absence of field."
Sure, that's pretty confusing, but it's the central focus of the poem. This guy really can't get over that wherever he goes, he's becoming an "absence". In fact, it bothers him so much, he has to keep moving. Ever known one of those people who can't seem to sit still? Voilà! Our speaker is a little like that.
So while the setting is a bit vague and unspecific, it's a symbol for all physical space. No matter where he goes in the world, the speaker feels this eerie sense of absence.