The Underground Man again addresses us, his audience: "You believe in a palace of crystal," he begins, a palace which cannot be destroyed and which is so all-around wonderful that no one could put his tongue out at it. (He's going to run with this tongue-putting-out image, so just remember that sticking out your tongue at the palace = mocking the palace.)
And we start down a slightly convoluted path of reasoning. Let's suppose the Underground Man were caught in the rain and took shelter in a hen house. As grateful as he was to the hen house for keeping him dry, he wouldn't call it a palace simply out of gratitude.
Why? Isn't the hen house just as useful in the rain as a mansion?
Not, the Underground Man says, if you have convinced yourself that the object of living is to live in a mansion. Once he's decided that, he won't be satisfied with a hen house, no matter how useful.
Make sense? OK, now let's suppose there is no such thing as a mansion, but the Underground Man still wants to live in one. He will never be satisfied with all the hen houses in the world, because he'll be searching for his mansion that doesn't exist.
This extends to the idea of the Crystal Palace. Even if it is a completely imaginary dream, even if it's impossible to have an actual Crystal Palace, the man who believes in the Crystal Palace will never be satisfied with anything less.
The point he makes is that once a person has an ideal, he won't compromise it – not until you show him something better. And until someone does show him something better, he'll just stay underground, thank you very much.
Back to the Crystal Palace. The Underground Man is adamantly against the palace for the sole reason that he can't put his tongue out at it. And it is the only building he can think of that he can't mock.
The thing is, he doesn't even like putting his tongue out at stuff. Or at least, he wishes that he didn't like it. He hates that he desires to mock everything.
He starts rambling but interrupts to say that underground men like himself ought to be kept on street corners, because if they're underground for forty years (like he has been), all they do is talk and talk and talk when they come out.