"A Song of Despair" is all about the speaker feeling abandoned, desolate, deserted… uh, anyone have a spare thesaurus? He is reflecting on his long-lost love, and it makes him feel as lonely as the wharves after all the ships have gone. He even calls himself "Oh abandoned one" for goodness sake. Seriously. Who does that? It's not clear if the woman abandoned him, or if he just feels lonely after their breakup, but abandonment in this poem is the name of the game.
The wharves are used as a symbol of abandonment in the poem, but we know that the ships will come back. What we don't know is if the girl will come back to the speaker. (Especially after he calls her a "pit of debris"!)
Not so fast there. The speaker calls himself the "abandoned one" in the last line of the poem, but there is evidence in the poem that suggests that he's more the abandoner than the abandonee.