The tone of the poem shifts here, as the day ends. Yeats uses this imagery to connect time with negative things. The three become quiet and contemplate the effects of time on their lives.
A moon, worn as if it had been a shell Washed by time's waters as they rose and fell About the stars and broke in days and years. (32-34)
Time has smoothed out all the moon's edges, like the waves do to a seashell. The imagery of time as "breaking" like waves makes it seem inevitable, even a little scary. Time has a lot of power in the poem, that's for sure.
That it had all seemed happy, and yet we'd grown As weary-hearted as that hollow moon. (38-39)
Time not only made the moon smooth, it also made it hollow. It also made our speaker, and the object of his affection, weary-hearted. No wonder he isn't a fan.