The only family that Harry has—the Dursleys—are pretty awful. His Uncle Vernon can't stand him, his Aunt Petunia does nothing to protect him against her husband's cruelty, and his cousin, Dudley, is free to bully him.
If we lived with these Muggles, we'd hate summer vacation, too.
While the Dursleys have made it clear that they resent Harry's presence in their house, they have upgraded him from his room in the cupboard under the stairs:
UNCLE VERNON: I should think you'd be a little more grateful. We've raised you since you were a baby, given you the food off our table, even let you have Dudley's second bedroom purely out of the goodness of our hearts.
Oh, yes. Harry's so very grateful. We get the feeling that the Dursleys have only done this because they're kind of afraid that Harry might jinx them with his magic wand if they don't at least pretend to treat him decently.
But when Dobby drops the cake on Mrs. Mason's head and costs Uncle Vernon his big business deal, this is the final straw for the Dursleys. They lock Harry in his bedroom and announce that he's "never going back to that school" and he's "never going to see those freaky friends of [his] again." All we need to hear is a maniacal laugh.
Really, when you think about it. This approach doesn't make a lot of sense. If Harry is such a thorn in their side, why not let him go to Hogwarts. Heck, drop him off early. In a way, Ron breaking Harry out of the house on Privet Drive is the best thing for everyone. Take a nice nine month long break and then get back to hating each other next summer, guys.