Our speaker is a sad and confused dude, who may or may not be Thomas Hardy himself. Sure, Hardy's experience of his wife's (Emma's) death was painful, confusing, and full of mixed emotions (check out "In a Nutshell" for more deets on Hardy's relationship with Emma), but the speaker of a poem, even an autobiographical one, is always a kind of fiction. We have no way of knowing how truthful a portrait of himself Hardy made, which is why we refer to "the speaker" of the poem instead of Hardy.
What we do know about this speaker is that he's having trouble telling fantasy from reality. He's hearing the voice of a mysterious dead woman, but is the voice just the wind? Is he reading his own feelings and senses into the outside world? Or, is there really a ghost in this poem? These are the heavy questions that the speaker is asking himself in "The Voice." Our speaker is one mixed-up fellow.
And you know what? We're right there with him. At the end of the poem, the speaker is unable to come to a conclusion about the true nature of the voice, so—just as we're drawn in to his confusing, jumbled sensations in the poem—we're left without any degree of certainty. Thanks a lot, speaker!