So, we'll go no more a roving So late into the night, (1-2)
In a poem that talks a lot about death, the line "late into the night" takes on a new meaning. Sure, it refers to the time when the speaker likes to rove, but it also sounds like he feels he's "late into the night" of his life.
For the sword outwears its sheath, (5)
When a sheath gets worn out, it pretty much dies. This very clearly tells us that the speaker is thinking about death. It's not clear whether he means the death of his enjoyment, or his own death.
And the soul wears out the breast, (6)
The confusion of line 5 is cleared up a little bit. Any image of the soul wearing out the body, or leaving the body, is a reference to death. The speaker is thinking about the death of his body and his soul's final journey.