I sent thee, late, a rosy wreath, Not so much honoring thee, As giving it a hope that there It could not withered be.
The speaker stops talking about thirst and drinking. Now he moves on to tell a little story about his relationship with the girl.
He says he sent her a "rosy wreath." Why? "Not so much" because he wanted to show her how much he likes her. Instead he wanted to give "it" (the wreath) the hope of everlasting life ("it could not withered be").
In other words, the speaker views Celia as some sort of divine or enchanted figure that can keep things alive that will normally wither and die (like a wreath of flowers).
But thou thereon did'st only breathe, And sent'st it back to me;
Hmm. Seems like the speaker's experiment didn't quite turn out as planned. That "but" at the beginning of line 13 tells us as much. The speaker basically says, "rather than keep the wreath to see if it wouldn't die, she sent it back to me."
Apparently, the woman breathed on the wreath before she sent it back.
Since when it grows, and smells, I swear, Not of itself, but thee.
The speaker knows that the woman must have breathed on the wreath "since" when it grows it smells, not like a wreath of flowers, but like Celia.
The phrase "I swear" is a bit odd. It sounds like the speaker is saying, "I swear it smells of thee," but that is awkward because then we are left with an incomplete clause. For it to make sense, the speaker would have had to say, "since when it grows, [it] smells, I swear, not of itself, but [of] thee." Of course, it is possible that the speaker is engaging in the age-old poetic practice of leaving things out.
Alternatively, the speaker could just be saying, "I swear," like when we say "it's true, I swear." Either way, it's pretty clear what the lines mean.
It's also important to note that the wreath still "grows." But wait. Don't you have to cut the flowers or plants to make a wreath? So how could it still be growing? Well, it seems like the speaker's wish from line 12 has come true (the wreath hasn't "withered"). Still, we bet he wasn't hoping to get the wreath back.