Sir Peter (Ella's father)
While he does have a few good lines, and he's not actively violent against Ella, this man is a selfish jerk, which does negatively impact Ella. He sends her to finishing school and marries Dame Olga, indifferent to the effect being near Hattie and Olive has on Ella (which is detrimental, to put it nicely). He also tries to marry off Ella to an older rich dude, and the only reason it falls through is that rich dude is not rich enough for Sir Peter's tastes. Oh yeah, and then, once he loses all his money, he still tricks Dame Olga into marrying him.
Stay classy, Sir Peter.
Okay, so she doesn't pull the same wicked stepmother act that other fairy-tale wicked stepmothers do. It's not like she tries to kill Ella. She just tries to make her life miserable, especially after learning that Sir Peter pursued and married Olga while poor. Taking her anger at marrying a poor man out on his daughter definitely puts Dame Olga in the villain category.
Perhaps the cruelest character in the book, Hattie not only threatens Ella's well-being on the journey to finishing school by starving her, but she also emotionally attacks Ella repeatedly by ordering her to not be friends with Areida and to not see Char. Some of the motivation for these attacks is undoubtedly selfish, as Hattie would like to win Char's heart instead, but as far as we can tell, Hattie is just a mean-spirited person. Pure antagonist.