In Chamber of Secrets, we find a number of examples of people fearing others who are different from them. The Dursleys fear Harry's magical abilities, so they lock him in a room by himself to keep him hidden from the rest of their neighborhood. The rest of Hogwarts fears Harry because he can do things they can't: he can speak Parseltongue, a Dark gift… and he seems to have a real talent for finding trouble.
We also see fear rising out of acts of what is essentially terrorism. The basilisk's attacks on Hogwarts students create a panic which causes divisions. Ravenclaws, Gryffindors, and Hufflepuffs all fear and/or hate Slytherin House, because nearly all Dark wizards come from Slytherin. And most students fear Harry because they think he is the Heir of Slytherin. Once the panic is started, there appears to be little or nothing that anyone can do to stop it. That's the core of Voldemort's power: he's a terrorist, in the literal sense that he spreads terror. His manipulations are successful enough that he doesn't even have to be there (or alive) to turn decent wizards and witches against each other. Voldemort's terrorism is present in Chamber of Secrets, but it really comes out in Book 6 and 7.