(Click the character infographic to download.)
We've said before that we don't understand Uncle Vernon, but we really don't get Aunt Petunia. Harry is her dead sister's child, whom she has raised since he was a year old. Yet she's perfectly willing to treat Harry like a slave. When the Masons are scheduled to arrive for their dinner party at the Dursleys, Aunt Petunia sends Harry out to work in the garden in the hot sun all day. When Dudley runs inside complaining that his cousin has been doing magic, Aunt Petunia "knew [Harry] hadn't really done magic, but [Harry] still had to duck as she aimed a heavy blow at his head with the soapy frying pan" (1.85). Aunt Petunia also does nothing to try to protect Harry when Uncle Vernon locks him inside his bedroom and puts bars on his windows. Clearly, Aunt Petunia loathes magic to a crazy degree, but we have to wait until much later in the series (try Book 7) to find out why.
Of course, one reason why Aunt Petunia tries to repress Harry's magic as much as possible is that she has a lot of faith in the importance of being Muggle (i.e., "normal"). About anti-wizard prejudice on the part of Muggles, J.K. Rowling has commented:
The wizards represent all that the true "muggle" most fears: They are plainly outcasts and comfortable with being so. Nothing is more unnerving to the truly conventional than the unashamed misfit! (source)
The Dursleys are "truly conventional," and Harry is growing into an "unashamed misfit," or in other words, a wizard. No wonder they don't get along.