The title of the poem, "They Flee from Me," is also the first line of the poem. Of course when we first read it, we're not sure who "they" are, and why they're fleeing from our speaker, but don't worry, we'll figure it out soon enough.
And indeed we do. In the second stanza, it becomes clear that the "they" of the title refers to the women. The speaker is puzzled at the fact that he is no longer a total stud. The ladies just don't come around like they used to (they flee).
Flee is just about the perfect word choice. Really, it's awesome. Why? Because for the whole first stanza, and even some of the second, our speaker is comparing these lovely ladies to wild animals, who "range" and "take bread" at his hand. Fleeing totally fits with all this animal behavior. It's as if these women have spotted a lethal threat (that we can't see) and their fight-or-flight response has kicked in. Their choice? Flight, of course.