The narrator is flat-out terrified of the empty pants. (So are we, to be honest.)
For starters, they are empty, are hued ghostly green, and trot about on their own. That's pretty scary in and of itself, but they also seem to be gunning for the narrator. They almost run him down with a bicycle (Scared.29), they row right for him (Scared.40), and they corner him in the Snide bushes (Scared.66-69).
But the whole thing is just a big misunderstanding. The pants weren't gunning for the narrator or trying to hurt him at all. In fact, they were just as scared as he was about the whole ordeal. The narrator could only see the situation from his own point of view, and it isn't until the end of the story that he (and we!) realizes the truth.
If you want to get fancy, you can say that the emptiness of the pants symbolizes the emptiness of the threat. (See how we did that? Okay, we give Seuss some credit.) Bottom line: once you understand and know the threat, it might not actually be that scary.