The specter of Peter Quint is the most obvious antagonist here. He's a kind of embodiment of evil itself; the Governess's descriptions of him make it clear that he emanates some sort of malevolent force. If we are to believe everything our narrator tells us, then Quint (or Quint's ghost, to be more precise) is behind all of the things that go wrong in the story. We're meant to understand that he corrupted both Miss Jessel and Miles, and that his manipulation of Bly's inhabitants from beyond the grave is what drives the action of the story. Notably, Quint is also of a lower social standing than everyone else, except Mrs. Grose, which makes him doubly threatening – the fear here is not just of adults (or their ghosts) corrupting children, but of social inferiors corrupting the elite.
Miss Jessel is a little more complicated than her male counterpart (yes, we know, you're probably sick of seeing the word "complicated" in relation to this story by now). We're not exactly sure if she's to be feared or pitied, or both. Either way, she's certainly not a positive presence, to put it lightly.
This is where things grow more complicated. The way the Governess tells it, Miles and Flora are ultimately both in cahoots with the two ghosts, and are therefore antagonists – but, at the same time, she wants to save them from their evil ways. We're often alerted to the possibility that the children know about the ghosts and simply don't tell the Governess and Mrs. Grose about them; this idea of the children being corrupted by the demonic figures of Quint and Miss Jessel, both before and after their deaths, is the most horrifying thing.
However, let us not forget that we're not entirely sure about how much we can trust the Governess – just because she's our narrator for the bulk of the story doesn't mean that she's entirely honest or sane. There's always the possibility that these roles are reversed: in another reading of the story, the Governess could be seen as the antagonist, and the children as the innocent parties. We'll never know for sure what James intended us to see.