The setting that we really care about in Harry Potter is, of course, Hogwarts School for Witchcraft and Wizardry. Privet Drive is simply nightmarishly normal and Mugglish. There’s nothing fantastic about the Dursleys' home at all.
By contrast, Hogwarts Castle – which is only accessible through a magic train we Muggles can’t even see – is filled to the brim with giant squid, trick staircases, Hungarian Horntails, and Forbidden Forests. Every corner of the castle seems to have some new enchantment to discover. It’s like a dream come true for everyone who has ever been bored with humdrum Muggle life or school.
But along with the living portraits and Quidditch pitches comes real danger and emotional agony. Just because the magic world is cool to read about doesn’t mean that it would be safe or pleasant to live there. A lot of social problems that we endure in our lives persist in the wizarding world, including poverty, racism, and terrorism. J.K. Rowling sums up:
Harry entered this world that a lot of us would fantasize would be wonderful: “I’ve got a magic wand and everything will be fabulous” – and the point being that human nature is human nature, whatever special power and talents you have [… Harry] walks into this amazing world, and it is amazing, and he immediately encounters all the problems you think he would have left behind and they are in an even more extravagant form because everything is exacerbated by magic. (source)