As gardeners grow, this one's of the grumpy variety. Crampfurl is Great-Aunt Sophy's lump of a gardener and Mrs. Driver's closest confidant. We don't really get to see Crampfurl in action—all our knowledge of him comes from what Arrietty overhears from her home under the clock. But still, we know this guy's a jerk, because he's in cahoots with the villain of the day: the dreadful Mrs. Driver.
We know that he and Mrs. Driver hang out together after hours, drinking Great-Aunt Sophy's wine and shooting the breeze (by which we mean gossiping). So it only makes sense that he has her back when she sees the little people:
"I saw them, I tell you," gasped Mrs. Driver, "little people like with hands—or mice dressed up…"
Crampfurl stared into the hole. "Mice dressed up?" He repeated uncertainly.
"Hundreds of them," went on Mrs. Driver, "running and squeaking. I saw them, I tell you!" (17.10-12)
Okay, so when we say he has her back, what we really mean is that the dude doesn't write her off as totally off her rocker. And that would be so easy to do. After all, she's talking nonsense here, and Crampfurl, though grumpy, is not dismissive. So that's something.
It's Crampfurl who first gets suspicious of the boy's odd behavior—he watches him calling down badger holes in the fields—and suspects him of hiding something.
But the poor gardener ain't so smart. So he never puts two and two together to realize the boy is trying to get in touch with Arrietty's borrower family. He just thinks he has a pet: "but I'd see'd him all right and heard him. Calling away, his nose down a rabbit hole. It's my belief he's got a ferret" (12.17).
Ugh. In this moment, you can't help but feel a little bad for the guy, right? The way Crampfurl talks ("I'd see'd him all right") hints at his lack of education and social standing. And really, his best friend in the world (as far as we can tell) is Mrs. Driver, of all people. Things are not looking bright for this man's future.