When we meet a human character in Watership Down, that's the question we ask ourselves: "are these humans mean, stupid, uncaring, or some combination of the three?" Because from a rabbit's perspective, humans are dangerous monsters. Honestly, if you're a rabbit, it doesn't matter whether the human is mean, stupid, or uncaring, because the result is almost always the same: the human is going to kill the rabbit.
In fact, we can count on one hand the number of times that rabbits and humans have good interactions: (1) There's the "kind old schoolmaster" (34.4) who rescues an orphaned Woundwort and feeds him with an eyedropper; and (2) there's the smart, kind young Lucy Cane who rescues Hazel from the family cat and takes Hazel to Doctor Adams.
Against those two examples, we have a legion of people are dangerous to rabbits: Woundwort's family being shot and dug out by a cottager; the Sandleford Warren being poisoned and dug up; Hazel being shot by farmers; the farmer who feeds and traps Cowslip's rabbits; etc. Even when humans aren't actively trying to kill rabbits (sometimes it's just an oopsy), a rabbit dies all the same. For instance, the Efrafan Charlock and some other rabbits get killed by a train (34.13). The people on the train aren't trying to kill rabbits, but—whoops!—a rabbit still dies.
Which is why the narrator makes a reference to rabbits and "their enemies—weasels or humans" (5.14), lumping humans in with any other predator. And except for a few characters—the schoolmaster, Lucy Cane, Doctor Adams—we can't really blame rabbits for fearing humans as much as or more than they fear weasels.