This poem takes place at home, in bed—lazy bones style. It's pretty much the most appropriate setting we can think of for a celebration of lovers. As we find out in the second-to-last stanza, the couple hasn't left the house all morning: "It's almost noon, you say? If so/ Time flies" (21-22). They're not in a hurry to go anywhere.
In the first half of the poem, we do get little mini departures via the speaker's imagination: to the library ("Turning some liver-spotted page"), the mall ("rising in an elevator-cage"), school ("listening to a bleak/ Lecture"), and a garden ("planting a raucous bed/ Of salvia"). Still, he never lingers in any place too long. The real setting is in the lovers' lair.