We're not sure how long ago this love affair took place in "A Song of Despair," but it's definitely over. The whole poem starts with a memory, and the speaker can't shake it. Almost all of the verbs are in the past tense, and the woman the poem is about is definitely no longer in the speaker's present. His sad, sad memories are all that's left. Still, he clings to them in such a way as to suggest that he's unable to let go of sad days gone by. In a way, we think that makes this poem somehow even sadder. Who knew that was even possible?!
The speaker is trapped in the past, which is the real shipwreck in this poem. Live in the now, man!
The past is almost like a place in the poem, a place the speaker longs to return to, but—sadly—has been banned from.