In the dystopic world of Gilead, no one can be trusted; everyone is a potential enemy. Some people, like Aunt Lydia and the Guardians, or the Eyes, are obvious antagonists, but they may not actually be the most dangerous people to the narrator. At least she can tell what side they're on just by looking at them. This isn't true for more everyday individuals like Ofglen, the handmaid down the street, or Nick, the chauffeur. Are they spies? Are they working for the resistance or The Man? It's impossible to tell – at some point, you just have to have faith, and faith is one of the things that gets crushed in Gilead. The narrator can't trust the people whose house she's in, either. Both the Commander and Serena Joy command her, separately, to endanger her life by breaking the rules. As the narrator says, her life would eventually be in danger anyway, but they speed up the process.
Even family can potentially be antagonistic. Left to her own devices for long enough, the narrator questions her husband Luke's love for her, and whether some part of him might have enjoyed being a man in the new world of Gilead. And when the narrator is finally lucky enough to see a picture of daughter, she can tell the child has no recollection of her. It's not the little girl's fault, but it's a betrayal that hurts just as much, if not more, than the others.