For my first twenty years, since yesterday, (line 1)
The first line of the poem sets up a blatant contradiction: the poem's "metaphysical conceit." The conceit is that while time passes normally for everyone else, it feels really long to the speaker.
Tears drown'd one hundred, and sighs blew out two ; (line 5)
Line 5 is the point at the poem when it becomes obvious that no one could possibly live this long. Time passes faster and faster, which really means that time feels longer and longer for the speaker.
A thousand, I did neither think nor do, (line 6)
Time makes another leap, jumping from the hundreds to the thousands of years.