Before he came to Japan, Rodrigues heard horror stories about Inoue, magistrate of Nagasaki—and one of the most fearsome men on the planet. He's basically Dr. Doom without the mask: a powerful leader willing to commit unbelievable horrors to retain his power.
So, with this in mind, you can understand why Rodrigues is surprised to learn that this supervillain is nothing more than an old man who seems pretty kind.
Unlike his fellow government officials, Inoue doesn't hate the Catholic Church. To him, it's purely a practical issue. He likens Portugal, Spain, England, and Holland to potential wives for Japan: each country offers trade opportunities and political alliances that could benefit his country. The Catholic Church, on the other hand, is an "ugly" and "barren woman" who has literally nothing to offer—besides some extra stress (7.28). Though he holds no real hatred towards its teachings, Inoue opposes Christianity for this reason alone.
Still, that's not going to stop him from getting straight-up medieval in his tactics against the Church. In fact, Inoue's emotional detachment from the issue allows him to cook up some of the most gut-wrenching torture techniques ever—like that pit where Japanese Christians are suspended upside-down and slowly drained of their blood. But the fact is that, politically speaking, Inoue doesn't even care about those folks—they're just "small fry" used to force "the fathers to apostatize" (7.100). To him, the Japanese Christians are mere pawns in chess game being played on an international stage.
In the end Inoue is just another powerful person willing to hold on to his power by any means necessary. Though he might not be intimidating to look at, you've just got to trust us on this one: this old man packs a mean punch.