The speaker of this poem is elusive. He (we say "he" only because Frost is male and that makes things a whole lot simpler) speaks without using the first person and keeps the topic pretty impersonal. Maybe he's the kind of guy who likes to make observations about the world and keep his personal life private. Actually, that sounds a whole lot like Frost.
In fact, we kind of think that's the point. Sure, we could imagine all kinds of stuff about the speaker (maybe he's an old guy, reminiscing about the bygone days of youthful vigor… or something), but the poem itself doesn't need a speaker. It sounds like a proverb, or an aphorism, or an adage. It sounds like something we recite, and say to each other, some wisdom that we pass on through the centuries.
And that helps it ring truer than if we tied this poem to some lonely old dude staring at the leaves outside his window. This poem doesn't need a speaker, because it has one in us.