How we cite our quotes:
That time of year thou mayst in me behold,
When yellow leaves, or none, or few, do hang
Upon those boughs which shake against the cold, (1-3)
This poem doesn't waste any time in making it clear that time is the major theme. Heck, it's even the second word of the poem—which is pretty much as close to the beginning as it can go and still make sense. Time is introduced right off the bat—but what does the speaker think and feel about time? Clearly, Shakespeare doesn't waste time in this department either. Right away in lines 2 and 3, we get vivid imagery telling us that the speaker is none too pleased with the way time is passing him by.
Bare ruined choirs, where late the sweet birds sang. (4)
Coming after what we've just looked at, line 4 might seem like just more of the same depressing imagery. This wouldn't be totally off the mark—it's certainly still meant to be a downer—but it would still miss the point. That's because line 4 actually introduces an important new idea: that the speaker used to be young and energetic—or at least that's what we think the reference to how the "sweet birds sang" is meant to suggest. What's more, those days of youth weren't even so long ago—as we can tell this from the word "late," which is just an old-fashioned way of saying "lately." That's just like time, isn't it? It shows up at your party, eats all your food, and then leaves without even saying "Goodbye."
In me thou seest the twilight of such day,
As after sunset fadeth in the west,
Which by and by black night doth take away,
Death's second self, that seals up all in rest. (5-7)
These lines act out the relentless process of time, as the past (the day) becomes a memory in (the twilight still left after the sun has set), and that memory is eventually obliterated by death. The phrase "by and by," linking up by alliteration to the B sound in"black night" paints a vivid sound-picture of the steady, minute-by-minute process of time's destruction.