Love is the only thing that counts for the speaker. He really fell for Annabel Lee. Even when she was alive, loving her was the only thing that mattered to him. He says she felt the same way, but of course we don't get to hear her side of the story. Actually, women don't ever talk very much in Poe's poems and stories, they just hang out and look beautiful and spooky. It would be sort of sad if it turned out that she wasn't that into him after all.