They call James Miliam the "meanest-looking man in Pittsylvania County, Virginia," and for good reason—to start, he has facial hair that puts that beard bowl guy to shame (2.6.2). But don't take that to mean that he doesn't have a soft side.
That soft side is named Martha Logan, mother of Nanny Delany. Martha is a rarity in her time: a strong, intelligent woman of mixed racial descent living as a free woman in the South. Despite the fact that she would be much safer in the North (or Canada, for that matter), Martha is far too proud to leave her home out of fear.
Luckily, she's got James Miliam there to lend a helping hand. James makes no bones about the fact that he'll demolish anyone who harms Martha, and everyone takes him at his word. We see what might've happened if someone messed with her after James tells a dinner guest that he "might soon find himself dead and shoveling coal for the Devil" for speaking down to his black grandson (2.6.25). If anyone laid a hand on Martha, we're sure that things would get even nastier.
But you'd be dead wrong if you think that this is a relationship between a strong man and a weak woman—to the contrary, Martha is portrayed as strong and intelligent in her own right. If you were hanging around Pittsylvania County at the time, you would often see "this colored woman just bossing around the fiercest-looking white man" (2.6.22). She evens stands up to him when he says something racially offensive, and he loves her all the more for it.
The kids realize that "Mr. Miliam would not last long without" Martha, and her death proves them right (2.6.26). Although he's one tough dude, he only wants to be tough for the sake of the woman he loves. Yes, we know, this is The Notebook-level romance. The love between this white man and black woman—a love as committed as that of any marriage, despite the laws preventing them from making it official—keep Sadie and Bessie from resenting white people too much.
After all, they've known a white man who was ready to kill to protect the people he loved, regardless of race. That's powerful, son.