Our speaker in "They Flee from Me" definitely has some abandonment issues. He says that a lot of women used to visit him, but they don't anymore. He's upset, miffed, and, frankly, a bit puzzled. Yet while the speaker plays the role of the victim, he also suggests that he, too, has abandoned plenty of women. This sure gives us an interesting glimpse into the sexual politics of 16th century England.
The real problem our speaker faces is that he is jealous of the sexual successes of the women in the poem. He feels left behind as they grow ever more promiscuous, and leave him for greener pastures.
The speaker unfairly suggests that a woman abandoning a man is the equivalent of being uncivilized: instead, she should be a "tame," functioning member of society.