Oliver, Isobel, Will, and Edmund are Arthur's stepchildren. They're the family that he and Stella never got to have, and he obviously loves having them around. But even so, they can never really understand what he's been through:
This was a sport, a high-spirited and harmless game among young people, for the festive season, and had an ancient tradition, too, as Will had rightly said, there was nothing to torment and trouble me, nothing of which I could possibly disapprove. (1.46)
And he wants it that way. When they start telling ghost stories, they have no idea why it would bother him so much, and he doesn't want them to know. We want to protect our families from the horrors of the past—and we're left wondering at the end if Arthur truly has exorcised his demons, or if they're still lurking around in the wings, just waiting for the perfect moment to destroy this fragile happiness.