Where It All Goes Down
On the surface, at least, the setting of this poem is something we know a ton about. In a sense, this whole poem is about describing the setting. We start out by learning that this poem takes place in a green valley (the greenest, in fact). There's a beautiful palace in that valley, with yellow banners flying from its roof. On top of that it has sweet-smelling walls, two shining windows, and a door made of rubies and pearls. The feeling we get from this setting (at least at the beginning) is one of beauty, peace, and relaxation.
Of course, that doesn't last. In the last third of the poem (the final two stanzas), the poem shifts. We're in the same place, in one way, but in another way we really aren't. Suddenly the palace becomes a sad, grim, "pale" place, with red windows and hideous creatures boiling out of it.
Importantly, through the poem's use of allegory, the setting here is all-important. It's not just a palace we're talking about here, it's a human head. And really, we're not just talking about a human head that sits in a jar in some creepy scientist's laboratory (which he'd probably pronounce "lah-bore-ah-tore-ee"). It's more like a person's psychology, their inner mental state. So, when we're seeing how the setting—one nice and peaceful—is getting overrun by evil gremlins, we're really looking at the onslaught of madness in the human mind.