Tom is a lot like another villain in the book—Jolly. He's a classic bully who's willing to throw racial slurs at Manjiro without any sense of guilt. On the first day of school, he tells Manjiro:
"I don't think you're going to fit in here. I don't think we let people like you into this school […] Go home, little slant-eyes… Go home where you came from." (3.19.28-29)
Nice guy, huh?
But like Jolly, there's more to Tom than the big bad bully persona he gives off. When Manjiro loses to Tom in their horse race challenge, everything seems to be going Tom's way… until his dad interrupts Tom's celebration in front the audience, berates him for taking the horse without asking him first, and then beats him. Oh.
Manjiro observes that, "although everyone had thought the black and blue marks [Tom] always wore had come from fighting, they probably were given to him by his own father" (3.22.70). Not that all the abuse he receives excuses Tom's bullying behavior, but it does give us an understanding of where all his violence and anger stems from. And for this reason, it's hard not to feel badly for him, especially when Manjiro spies him in a ditch, bloody and crying. Tom may act tough, but he's a wounded kid when it comes down to it, both physically and emotionally.