We're going to go out on a limb here and say something you'll probably never hear Shmoop say again: the speaker doesn't matter much here. K, thanks, bye!
Oh, we should probably explain ourselves. It's not that the speaker doesn't matter at all, it's just that in a Wallace Stevens poem, the speaker—whoever he or she is—will always always always take a backseat to the ideas in play.
You can think of the speaker as a detached, omniscient narrator of sorts. He's not stuck in any particular person's brain, so he uses the word "one" to describe, well, anyone. And that's because he's talking about everyone.
Imagine if this poem said "I must have a mind of winter" or "She must have a mind of winter." And instead of "For the listener, who listens in the snow," it said, "For Shirley, who listens in the snow." That's quite a different poem, and quite a less effective one if we may say so ourselves (and we always do). Part of what makes this poem so successful is that we don't have to worry about the who—just the what. And the what's difficult enough to keep us occupied for days.