As is so often the case, it's really easy to imagine Morgan Freeman speaking these lines, or at least to his character from The Shawshank Redemption. In that movie, he plays a prisoner who made a mistake early in life and has spent the last 50 years in prison. He realizes he screwed up, but he refuses to let prison prevent him from having something resembling a life. As his tells his fellow prisoner: "get busy living, or get busy dying."
That's almost exactly the message of this poem: take advantage of your opportunities or spend the rest of your life tarrying (not getting anything done). We imagine the speaker, like Freeman's character, as an aged man who perhaps didn't gather his rosebuds while he could but is trying his best to gather them now. He speaks with authority and he'll be damned if those virgins are going to repeat the same mistakes he made. It's almost like he knows from experience that "worse, and worst / Times still succeed the former."