Where It All Goes Down
The speaker of "The Voice" projects his feelings onto the landscape. He's feeling sad inside, and what he sees outside of himself reflects that sadness. When poets reflect a person's inner life into her outer life (like the speaker's sadness into the landscape), we call this the "objective correlative."
T.S. Eliot, everyone's favorite modernist poet, came up with this fancy term, but don't let it scare you. It basically describes exactly what's going on in "The Voice." The speaker is grieving, and the world around him—especially the wind—reflects how he's feeling inside. Depending on your interpretation, the speaker may or may not project the woman's voice in his head onto the wind. And the speaker is clearly falling and faltering in his heart—so, hey, check it out, the leaves in the poem are falling!
The physical attributes of the setting, then, are essentially a reflection of the speaker's emotional state. And, as we learn particularly in stanzas 3 and 4, that state is a pretty confused and stormy landscape—both within and without.