They flee from me that sometime did me seek, With naked foot stalking in my chamber (1-2)
The speaker's female visitors don't just abandon him. They "flee" from him. This is a pretty strong word, as if he's either very dangerous or totally disgusting. Could this have something to do with the social consequences of having sex outside of marriage? Or are they just moving on to new and different men?
I have seen them gentle tame and meek That now are wild and do not remember (3-4)
Sometimes there's just no way to account for abandonment. The speaker describes women as "wild" and thus implies that there's no way to explain what has happened other than to assume that women are unpredictable.
But all is turned thorough my gentleness Into a strange fashion of forsaking (16-17)
The speaker says it is because of his "gentleness" that the woman forsakes or abandons him. The alliteration in "fashion of forsaking" emphasizes the act of abandonment, but also implies that this whole "forsaking" thing is just a fad, a "fashion" that won't last forever.
And I have leave to go of her goodness And she also to use newfangleness" (18-19)
While the speaker is probably being ironic when speaking of "goodness," the rhyme on "newfangleness" and "goodness" suggests that leaving things behind and seeking out new opportunities isn't always a bad thing.