This is certainly a good way of grabbing a reader's attention. The poem begins with a direct acknowledgement of mortality. And since this phrase is not only the title, but also the first line (which then gets repeated!), the poem really makes sure that we come face to face with it. That 'When' at the beginning also alerts us to the fact that death's approach is going to lead to something else – some act, desire, or train of thought.
takes all the bright coins from his purse to buy me (lines 3-4)
Aside from being an arresting image, this line sets up death as a sort of exchange. Whether the exchange is between death and the speaker (between death and each life it takes) or between death and another abstract (such as life in general) it's definitely an interesting idea. We see death personified fairly often, but usually he just takes a life. We've never seen him buy one before. Having a currency seems to suggest a system or balance. Can life buy the dead back from death, like how decomposition gives rise to new life?
I want to step through the door full of curiosity, wondering: what is it going to be like, that cottage of darkness? (lines 9-10)
We've already had some clues that our speaker has a pretty different understanding of, or approach to, death. But this line seals it. The darkness tells us that our speaker doesn't claim to know what goes on in death, but the image does give it a sort of location. We're used to thinking of death simply as an end, or as an event. Or else something that leads to heaven or hell. This line suggests that our speaker thinks it might be stranger and more mysterious than that. This is the completion of that 'When' and announces the main desire expressed in the poem: to approach the world (life and death included) with curiosity and a capacity for wonder, and without fear.
and I think of each life as a flower (line 15)
Flowers are notably short lived. So part of what this line does is to stress the evanescence of life. It also might suggest a connection between that brevity and the beauty and vibrancy that flowers also represent. Do we think flowers are beautiful partly because they're only around for a little while?
each name a comfortable music in the mouth, tending, as all music does, toward silence, (lines 17-18)
We could see this as another subtle way of recognizing that the beauty and splendor in life is tied to the fact that it is temporary. This line recognizes the loveliness of each name (and, by extension, each thing), by comparing it to music, which is a form of art, and one that is pretty much universally loved and appreciated. Then it points out that all music tends toward silence. What does all life tend toward? You guessed it! Just as music is defined by the silence around it, life is defined by death.
When it's over […] (lines 21 and 24)
Here, toward the end of the poem, we see that mortality has been rephrased. We began with "when death comes," and now we have "when [life's] over." Having life replace death as the center of this phrase seems to mirror the movement of the poem – beginning with a contemplation of death, it moves to a contemplation and philosophy on how to live.