Fagin is pretty clearly a bad guy. We're not quibbling with the fact that a dude who thinks, "Ooh! Million dollar idea! Let's turn little boys into crooks!" is basically wearing a sticker that says Hello, My Name Is Evil.
But the racial prejudice apparent in Dickens’s characterization of Fagin can make readers (super-duper) uncomfortable. Dickens often refers to him only as "the Jew," and a lot of traditional racial stereotypes against Jewish people are used: he’s miserly, has red hair, and is a corrupter of children. Who knew Dickens was an anti-Semite?
We have to back up for a moment here. For a long time (until very recently, actually), people thought that Fagin was based on a real guy who sold stolen goods (a.k.a. a "fence") named Ikey Solomon. Ikey Solomon happened to be Jewish, but the stereotype was there before Solomon or Fagin came along—the limited number of careers open to people of Jewish descent did indeed drive some Jewish people to illegal activity—but certainly the majority of criminals in London at Dickens’s time were still Christian. Dickens was really only a casual anti-Semite. He was no more prejudiced than most of his peers, and actually less so than most. But even knowing this, the level to which he allowed various anti-Jewish prejudices color his portrayal of Fagin still makes readers more uncomfortable than wearing a wool sweater in July.
Later in his career, Dickens tried to make up for the racial stereotype in his portrayal of Fagin—his last completed novel, Our Mutual Friend, has a Jewish character named Riah who is as virtuous as Fagin is villainous. As well, the 1867 edition of Oliver Twist changes a lot of the references to "the Jew" to "he" or to "Fagin." Does this make up for the anti-Semitism? No... but knowing something about where it came from makes it a little more comprehensible.
The final chapter about Fagin (3.14: The Jew’s Last Night Alive) shows how alienated Fagin was from the rest of society. And not just from society, but from the entire human race. He’s in a crowded courtroom, and is surrounded "by a firmament all bright with beaming eyes" (52.1). The crowd of people is reduced to this one feature: their "eyes" (the narrator doesn’t describe any other part of their body in that paragraph). So Fagin is made into a spectacle, and his own sense of individual identity is totally squelched by their "inquisitive and eager eyes."
In this scene, Fagin seems totally numb to what is happening to him, and he ends up watching what goes on in the courtroom "as any idle spectator might have done" (52.7). And later, when he looks into the crowd, "in no one face […] could he read the faintest sympathy with him" (52.3). So Fagin is out of sympathy with the entire mob here—no one can identify with him.
Of course, that’s not at all surprising, given how frequently he’s cast as sub-human, or rat-like, or demon-like. For example, right after he finds out about Nancy’s conversation with Rose and Mr. Brownlow, he "looked less like a man than some hideous phantom" (47.1), or when he’s in prison, when his face looks "more like that of a snared beast than the face of a man" (52.34).