Unwearied still, lover by lover, They paddle in the cold Companionable streams or climb the air; (19-21)
The swans are "unwearied," and the speaker, in contrast, seems very weary—tired, worn out, kind of sad. He's also alone, while these swans go "lover by lover." More sadness for our poor speaker.
Among what rushes will they build,By what lake's edge or pool Delight men's eyes when I awake some day To find they have flown away?" (27-30)
When the swans leave, the speaker will be deprived of something that "[d]elight[s]" him. Even though the speaker loves the swans, he also seems sad at the possibility that they might abandon him. In a way, even something beautiful and wonderful can cause sadness by its absence.