We like the image of clock hands spinning wildly out of control to describe this poem (see "Setting" for more on this). The poem dislodges us from normal earth-time and takes us into the world of the speaker's internal clock. The point seems not to be that time passes so much more quickly when his lover is not around. The point is that it feels like forever since the speaker has seen his lover, even though it has only been a day. The fact that the years in the poem add up to 100 times 24 should give you an idea of how important the Metaphysical Poets felt it was to be clever and witty in their writing.
The speaker's internal clock has slowed down to a crawl since he was separated with his lover, and he feels like they cannot be reunited soon enough.
The metaphysical conceit of prolonged time breaks down in the final two lines, as the speaker throws the traditional concept of linear time out the window.