The Dursleys are the Muggle family that has raised Harry since he was orphaned when he was one year old. Petunia Dursley was the sister of Harry's mom Lily, but she couldn't be more different from her kindly, magical sister. The Dursleys are as bigoted about wizards as the Malfoys are about Muggles.
The Dursleys absolutely hate to see Harry happy. When Harry gets invited to the Quidditch World Cup by the Weasley family, his uncle Vernon will only agree to it because Harry threatens to tell his murderous godfather Sirius if Uncle Vernon says no. And Uncle Vernon spends a lot of time upset in the early chapters of the book because he doesn't want to see or interact with the Weasleys in any way – he thinks they're freaks, and he doesn't want the neighbors to notice "[Harry's] kind" (4.21). To make matters worse, when the Weasleys come to pick up Harry, a string of misunderstandings leads them to wreck the Dursleys' living room. This just confirms all of the Dursleys' worst suspicions about wizards.
As revenge for the Dursleys' bullying ways, Fred and George Weasley "accidentally" drop a Ton-Tongue Toffee where the Dursleys' overweight son Dudley can get it. Dudley, of course, eats the toffee and suddenly finds his tongue growing to about four feet long. The Dursleys are so terrified by wizards that it takes Arthur Weasley ages to convince them to let him shrink Dudley's tongue for him. Arthur Weasley is furious with Fred and George, but they feel completely justified. Dudley is "a great bullying git" (5.17), after all.
Because the Dursleys have been neglecting and abusing Harry practically his whole life, it seems bizarre that Professor Dumbledore keeps insisting that Harry spend his summer vacations there. Clearly, he has a motive, but we don't know what it is yet. At the end of Goblet of Fire, even after Cedric's death has profoundly hurt Harry psychologically, he still has to spend at least part of the summer with the Dursleys – and surely no good can come of that.