Patricius and The Manichees
Watch out for bad father-figures. Patricius gets named an antagonist not just because he is a non-believer (after all, there are plenty of those), but also because he stands in the way of Augustine's conversion. Or, at least he doesn't guide him in the direction of Christianity when he should, which makes Augustine harbor some serious resentment for the guy
Speaking of bitter childhood resentments, you might assume that Augustine doesn't like his father because Patricius isn't the greatest husband to Augustine's mother. And we know what a mama's boy Augustine is. Patricius also does things like not care enough about his son's sexual proclivities, and care too much about his education, so Patricius is definitively a Bad Guy.
But some stories have more than one Bad Guy. Remember the politically incorrect Siamese Cats in Lady and the Tramp? If we're talking about a more generally consistent evil force in the novel rather than a single villain, it would have to be the Manichees. The Manichees are the ones who, for a long time, lead Augustine astray and fill his head with all sorts of false information. He has to unlearn all their teachings in order to complete his Hero's journey and find God.