Time has a wearying effect in "Adam's Curse." As the speaker considers the state of romance, writing, and beauty in society, he is reminded of his own love life and how it has, because of the passage of time, not worked out the way he probably intended. In fact, he says that the passage of time has worn out his heart, the same way the sea wears down a seashell. Smooth heart? Now that sounds painful. It's safe to say he doesn't have the most positive feelings about what time does to us.
Questions About Time
How does time operate in the poem? What does it do to objects and people?
What has changed over time, and what effect does time's passing have on the speaker?
Super-deep question: Is time the enemy of love? How would the speaker answer that question?
Chew on This
The speaker is nostalgic (or just grumpy) for the way things used to be. He's unable to cope with the passage of time, which seems to have done a number on him.
The speaker is confusing time passing with romantic rejection. It's not time's fault that you got shut down, buddy.