They flee from me that sometime did me seek,With naked foot stalking in my chamber (1-2)
The speaker's female visitors don't just abandon him. They "flee" from him. This is a pretty strong word, as if he's either very dangerous or totally disgusting. Could this have something to do with the social consequences of having sex outside of marriage? Or are they just moving on to new and different men?
I have seen them gentle tame and meekThat now are wild and do not remember (3-4)
Sometimes there's just no way to account for abandonment. The speaker describes women as "wild" and thus implies that there's no way to explain what has happened other than to assume that women are unpredictable.
But all is turned thorough my gentlenessInto a strange fashion of forsaking (16-17)
The speaker says it is because of his "gentleness" that the woman forsakes or abandons him. The alliteration in "fashion of forsaking" emphasizes the act of abandonment, but also implies that this whole "forsaking" thing is just a fad, a "fashion" that won't last forever.