Oops, sounds like something's changed here. Time is still personified as a him, and this time the speaker is concerned with "his tuneful turning so few and such morning song." What's up with that, time? What was a merciful bounty has become something that "turns" and only has a "few" morning songs. The speaker is beginning to experience the passing of time and it's not like he thought it would be.
This is also the section of "Fern Hill" that might allude to the Pied Piper of Hamelin. The Pied Piper played a flute that led children away from their town. He was owed a debt and the town didn't pay, so he took the children as payment. In the same way, time is playing a tune that leads children away from youth. It lends a creepy note to a so far joyful poem.