While the speaker of "They Flee from Me" brags about his sexual exploits, he makes it clear that there was at least one girl who was special. She was really pretty and sexy, and sometimes it even seems to us that he might have loved her. For the most part, though, love is conspicuously absent from the poem, and it's this absence that makes it all the more relevant. It seems our speaker loves sex, not women, and maybe that's the source of all his problems.
Sometimes sex involves love, but "They Flee from Me" shows us that often it's really just a form of control.
Part of the speaker's problem is that he loves the woman in the second stanza. But because he's been so promiscuous in the past, he doesn't know how to treat her, which makes him nervous and confused.