Not surprisingly for a Lowell poem, we're not exactly in one place the whole time. In fact, we're not really even in the same time period the whole poem. For the sake of our sanity, let's say the poem takes place in South Boston in the early-mid 1960s, but the speaker takes mini trips (via reflection) back to the Civil War, the old South Boston Aquarium, and the dedication of Shaw's memorial. Oh, and we almost forgot, he takes small detours all over New England, and to outer space. Lowell does this to create a big picture of where we have been in the past, where we are now, and where we're headed.
Unfortunately, the present and the future don't exactly look bright. From the death and slavery of the Civil War to the decrepitude and sterility of the modern day, it seems that—no matter where or when we are in this poem—things are bad all over. Society just doesn't seem to be able to live up to its promise, and much of what was good in the past is being erased by disregard for those that came before us.
As Lowell warns, "The ditch is nearer." We're not entirely sure what that ditch is, but we're pretty sure we don't want to go there. Too bad, though. The various settings in this poem act as a constant reminder of the dire straits of humanity. Watch out for any holes in the ground, and bring your passport and, er, your time machine, because we're all over the map—and all signs point to a bad place.