Gaffer is not pleasant guy to be around. Not only does he makes his living by hauling dead bodies out of the Thames River (which: gross); he's so insecure about his lack of education that he forbids both his children from getting any education themselves (which is even grosser than being a corpse fisherman).
We can tell he's not a likeable guy from the moment the narrator first describes him:
He was a hook-nosed man, and with that and his bright eyes and his ruffled head, bore a certain likeness to a roused bird of prey. (1.1.12)
Dude looks like a hawk and needs a haircut. Not too flattering. Dickens keeps on with this bird of prey image for some time, probably because it does a good job of symbolizing the line that Dickens often draws between good and bad people. Some people in this world are predators, he suggests, and the rest (in other words, the kind and caring people) are prey.
When Gaffer finds out his son Charley has run away to get a good education, he stabs the air with his knife and says he never wants to see his son again. As he says to his daughter Lizzie,
Let him never come nigh me to ask my forgiveness […] Let him never come within sight of my eyes, nor yet with within reach of my arm. His own father ain't good enough for him. (2.6.152)
As we've mentioned, Gaffer is pretty insecure about his place in the world. He's constantly worrying about other people thinking they're better than him, and this makes him treat people horribly… even his own kiddos.