Let's face facts: this speaker is a bit of killjoy. He's bummed out about getting old, and he can't stop to admire something as beautiful and graceful as swans without making it all about himself and his impending mortality. Talk about a Debbie Downer!
Still, that's no reason to brush him off, or pass him by as he sits there on the park bench with his face drooping into his hands. With age, of course, comes wisdom, and it seems that the speaker has his fair share to spread around.
Let's start with his experience. The speaker's been coming to this water, counting the swans for nineteen years. So, he's got to know these birds inside and out. (That idea is supported in his confidence that there are 59, and not 60, swans before him.) And, what's he learned in all that time? Well, in short, he's learned appreciation. Fundamentally, he appreciates what these swans represent for him: youth, beauty, energy, love—passion, baby!
Now, think about that for a minute. Does he resent the swans? Does he pick them apart ("Sure, they may be passionate, but boy do they make ugly chicks!")? No. He simply, plainly appreciates them for what qualities they possess, which is especially difficult to do if you, yourself, don't also possess the same qualities. The speaker, in the twilight of his age, has every reason to be petty and jealous of these magnificent birds, but instead he provides a great example of how to appreciate the wider world around us—even if we can't relate to it.
And, in a subtle way, he also represents the power that such observation can produce. While it's easy to imagine him as under the spell of the swans' magnificence for much of the poem, we think that the way he abruptly changes the portrayal of the swans in the final stanza (from "clamorous" to calmly drifting on "still water") seems to indicate that perhaps these swans might be under his influence instead. As he admires their energy, they flap around all over. As he turns to thoughts of old age, the get quiet and drift. Is that a coincidence? Do these swans really exist in front of the speaker? Or could they be an elaborate symbol, wrought by his own artistic imagination? The way you answer those questions will determine just how you think about the speaker of this poem.