We know right where this poem begins: in the speaker's bed, late at night. Where it ends, however, is another matter entirely. You can almost see the thought bubbles coming up from the speaker's bed and floating out into the wide, wide world.
See, by the time we're in the second stanza, this poem is clearly settled in the Real World. You know, that world where some people have cash and some people don't – a world that's economically and socially stratified. There's no room for dreamy idealism, or any of the pensive nighttime thoughts that occupy the speaker's mind in Stanza 1.
And then, by the time we're at Stanza 5, we've zoomed out far enough to be able to consider the entire "race of man." In other words, we've moved waaaaay back. We like to think of it as a "Earth seen from the Moon" sort of view.
So, from one little bed to the entire world at a glance? It seems like setting just isn't this poem's most important focal point. Or perhaps it's important that the setting shifts – if only because the problems that our speaker finds remain the same. Whether you're in your own bed at night or watching the rich and famous (probably on reality TV) or looking at the Earth from the Moon, you're probably alone. Or at least you feel alone.
When we get right down to it, this poem centers itself on an emotional landscape: the homelessness of the human soul. The roving, searching heart has no home- which is perhaps why we move from setting to setting in this poem as well.