Who is our speaker? Sheesh, how would we know? This guy vanishes when he stands still.
But in all seriousness, it's hard to say. He likes small words, short sentences, few adjectives, and no drama. Maybe he's the quiet type, or maybe he's like a wise old sage who doesn't see the point in wasting words. In fact, the vagueness of his identity reinforces the poem's theme of absence and presence. He's like the invisible man of his own story.
We'll say he's a man, but there's nothing in the poem that suggests a gender for the speaker. We have no idea what he looks like, how old he is, where he is (aside from a field). In fact, this poem follows a minimalist style. In order to get down to the core of his "reasons for moving," the speaker tells us as little about himself as possible, but reveals a lot about how perceives his presence in the world.
A cool way to think about this is to say that the speaker is telling us what he thinks about how he feels, and how he feels about what he thinks. We couldn't write a biography of him to save our lives, but we can tell you he's preoccupied with feelings of isolation and absence. He's a deep thinker, meditative, and thinks of himself as someone who disrupts the unity of nature. Those are big themes for such a little poem, and our speaker knows how to capture it all in just a few short lines.