The manager wants to oust Kurtz, because he's driven by greed and ambition to move up in the Company. Plus, he's annoying and just might have something to do with the "accident" that his steamboat suffered.
Like the manager, the brickmaker wants only to climb up the corporate ladder. However, he's even shadier: his rank is lower, and he's willing to resort to even lower tactics to get what he wants. It's implied that he uses his silver tongue to weasel his way into the trust of his victims—Company members with powerful connections. Plus, there's all that devilish imagery—the "forked little beard" and "hooked nose" (1.56) that make us feel, um, a little uncomfortable. And antagonized.
But let's be honest. If the real definition of an antagonist is someone who gets in the way of the protagonist's goal, the then jungle is Marlow's real antagonist. He wants to be upright, honest, and British, but the jungle delays him, frightens him, and even ends up corrupting him. And then it almost kills him.