Well, if the title "A Song of Despair" doesn't clue you in, then you're in for a surprise when you see just how weepy this poem is. It's sad, sorrowful, and lamenting. The speaker has lost his love and he is grieving for it. You might want to turn on some blues while you read if you like your poetry soundtracks to be thematic.
The speaker seems to be sad that he lost his lover, but the way he describes their relationship makes it sound like they were sad when they were together back then, too. Lose-lose.
The woman is the source of all sadness in this poem. A-ha!