In Holden's opinion, "phony bastards" are not only his but the worlds' antagonists. Holden is spurred to action (…or sometimes inaction) by all the people he meets who are just putting on a show. Dr. Thurmer the headmaster, George from Andover, Lillian Simmons—basically everyone he meets is phony in one way or another.
Seeing the majority of the world's population as an antagonist can be an isolating perspective, so we're not exactly surprised that Holden spends most of The Catcher in the Rye alone and alienated.
Seeing everyone in the world as an antagonist is not only isolating—it's indicative of a clear character flaw. So while Holden sees everyone else as the antagonist, we as the reader can see that Holden himself is his own antagonizing force. He chooses to judge everyone he meets, he chooses to alienate, and he chooses to be alone. When you look at it this way, Holden is his own worst enemy.