Let us clarify: this book wouldn't garner nearly as high of a Tough-o-Meter rating if it weren't for the ridiculous slew of historical references Hugo throws all over the place. Seriously, this novel is like an encyclopedia of obscure French history.
But fear not: this novel is also a good exercise in letting go. You don't have to understand it every time Victor Hugo makes mentions the Duke of Burgundy. You don't have to know the difference between a Roman and a Gothic arch. You don't have to memorize all of the French monarchs and the years of their reigns. Really, the most you need to know is that most of them were named Louis—and you'd figure that out pretty quickly, anyway. Oh, and you don't need to know Latin either (be grateful for endnotes).
The information overload in this novel might put off a lot of readers, but we're here to power you through it. Don't worry if you don't get why Hugo devotes an entire chapter to one particular square in Paris, or if you don't get how Parisian architecture changed throughout the centuries. There are a ton of other little technicalities that you probably won't be familiar with in this novel, but the point is that if it's important, you'll be able figure it out as you go along.
The plot of the novel and the language Hugo uses to tell it aren't actually that difficult. Sure, it's no cakewalk (whatever that is), but the story moves along pretty quickly and keeps taking bizarre turns. It's kind of a page-turner, the characters are quirky, and there's a so much emotion and comedy and tragedy that you might actually find yourself blitzing through it.