We get to do some time-traveling in this poem. Lowell jumps around frequently and without warning. We begin in South Boston at the aquarium that's been closed and boarded up for what seems like a long time.
Then we move back in time with the speaker to when he used to visit the aquarium and gawk at the fish. When he snaps out of that daydream, he recalls a more recent past ("last March" specifically) when he witnessed the construction of what would become an underground parking garage. Included in that same scene is the Civil War memorial for Colonel Shaw (we'll get to him in detail later), and his all-black infantry.
Then Lowell sneakily segues into a little info about the Civil War, and later, the dedication at the memorial. He stays more or less on the track for a handful of stanzas: he talks about how New England continues to remember the Civil War in graveyards and tattered flags, but how that memory is diminishing.
Then he sort of pops back into the present, with a somewhat critical eye. He talks about World War II and how there are no commemorative statues for that war. To wrap up, he winds the image of Colonel Shaw and the old aquarium's fish together, and sort of superimposes them over slow-moving traffic in present-day Boston.