The Hollow Men have a bad case of "the Shadow." Like when you sit down to do your homework, and you can't bring yourself do open your book, you too can blame the Shadow. But the Hollow Men have it even worse. They can't even respond to their own emotions. The Shadow represents their cowardice and the failure of their will. They can't even look the "eyes" in, well, the eyes. They turn around and around like the wind and wait on the bank of a river.
The Hollow Men have been damned for their cowardice and weakness of will, symbolized by "the Shadow."
The Hollow Men are not morally worse than the "violent souls" of Hell, but they are more contemptible. They envy the violent souls for their ability to at least look goodness in the eye, even if only to spit on it.