It's easy to miss, but the level of all-seeing power (or omniscience) that Victor Hugo claims in this book actually changes as we move from the early chapters to the later ones. In the early chapters, Hugo steps forward and claims that he, the author, is the narrator of this story. We see this claim in passages like this:
In another essay he examines the theological writings of Charles-Louis Hugo, Bishop of Ptolémais, a great-great-uncle of the present writer [Victor Hugo], in which he proves that a number of pamphlets published in the last century under the pseudonym of Barleycorn are to be attributed to this prelate. (126.96.36.199)
In these early stages of the book, Hugo admits to having limited knowledge of his characters and only knowing about them through historical documents and stories. When talking about Bishop Myriel, for example, Hugo claims, "As to the truth in general of the tales that were told about the early life of M. Myriel, no one could vouch for it" (188.8.131.52). As the novel carries on, though, Hugo starts diving in and out of character's minds like a dolphin playing in the ocean. Just look at some of his early descriptions of Thénardier and Inspector Javert and you'll see a clear difference from this early description:
He was cunning, rapacious, indolent and shrewd, and by no means indifferent to maidservants, which was why his wife no longer kept any. (184.108.40.206).
In this case, we find Hugo much more willing to straight up tell us what his character's mentality is like, rather than leaving it to us to decide from our own observations. It's almost as if the longer the book goes, the more Hugo wants to make sure that we sympathize with the right people and the right ideas.