In Holden's opinion, "phony bastards" are clearly his – and the world's – antagonist. Holden is motivated and spurred to action (or sometimes, inaction) by individuals who are affected and false. Dr. Thurmer the headmaster, George from Andover, Lillian Simmons – basically everyone he meets is phony in one way or another. Seeing the majority of the world's population as an antagonist can be an isolating perspective, so we're not exactly surprised that Holden spends most of The Catcher in the Rye alone and alienated.
Come to think of it, seeing everyone in the world as an antagonist is not only isolating – it's indicative of a clear character flaw. So while Holden sees everyone else as the antagonist, we as the reader can see that Holden himself is his own antagonizing force. He chooses to judge everyone he meets, he chooses to alienate, and he chooses to be alone. When you look at it this way, Holden is his own worst enemy.