Yeah, there's a lot of death. But these are specific types of deaths: they're the deaths of innocent people, like Jennet Humfrye's son Nathaniel who dies in an accident along with his nursemaid and the driver of the pony trap, Keckwick Sr. It's no one's fault and the dead themselves are innocent, but Jennet goes insane with grief and begins wreaking havoc on other people's children. Innocent children. As Arthur asks, and Sam explains:
"You mean any child? A child of the town?"
"Any child. Jerome's child." (11.127-128)
The ghost of Jennet Humfrye is none too picky. She'll pretty much kill off whatever child is around, even if the child is innocent and has nothing to do with her own sad story. This culminates in the end scene, where Arthur watches on in horror as Stella and Joseph die. Talk about blameless: they never even knew about the woman in black.
It's not just the literal death of innocent babes that we're talking about here, though. It's also the death of Arthur's innocence:
For I see that then I was still all in a state of innocence, but that innocence, once lost, is lost forever. (4.22)
At the beginning of the story, Arthur is young and living in a world that he believes to be rational and good. He is cured of those innocent ideas when he meets the woman in black and she effectively ruins his life, despite his unwillingness to believe in her.
And (we think) that's the thing about the past: it's illogical. It hurts people who had nothing to do with it. It flies in the face of reason and rationality. Like nature, it's terrifying—and ultimately indifferent to any individual lives it hurts along the way.