Uncle Vernon is a violent bully. He is petty and abusive to Harry just because he finds Harry's magical skill to be offensive and freakish. In the first chapter of Book 5, we see Harry lying under an open window in the front garden so that he can overhear the Muggle news because otherwise, Uncle Vernon won't let him listen. How small-minded and cruel is that? And Uncle Vernon also tries to throw Harry out of the house when he learns that Harry might be drawing Voldemort to the Dursley household. He is completely willing to cast out his own nephew into danger, without a second's thought about what it might mean to Harry to be tossed out on the street. It's only the insistence of Aunt Petunia that keeps Uncle Vernon from booting Harry out.
So, Uncle Vernon is a pretty lousy excuse for a human being. He believes above all in order and normality, and anything wizard-related offends his sense of what is proper in the world. At the end of the book, when Uncle Vernon and Aunt Petunia unwillingly go to meet Harry at the train station to take him to Surrey for another summer holiday, the Order of the Phoenix confronts them.
Tonks tells Uncle Vernon that, if the Dursleys are horrible to Harry, the Order will hear about it. Uncle Vernon draws himself up: "Are you threatening me? [...] And do I look like the kind of man who can be intimidated?" (38.250-252). Moody replies, "Well [...] I'd have to say you do, Dursley" (38.253). Like most bullies, Uncle Vernon folds immediately. He's a lot of bluster and hot air, but when you get down to it, Uncle Vernon is a small-minded bully. He doesn't have the strength of character to stand up to grown-up wizards, and the Order of the Phoenix knows it.