As he leaves to pursue his frenemies, the Underground Man concludes that this is real life, and that it is indeed very different from his dreams about the "sphere of art" that we heard about before.
Once outside, he gets into a horse-drawn carriage (read: taxi), crumpling with shame over the fact that he just borrowed six more roubles, but he concludes that he can't ask for forgiveness, so he must slap Zverkov in the face.
(Note: A slap in the face is a way to challenge a man to a duel. So he's not just slapping him, although that would be an interesting choice for revenge.)
Then he debates whether or not to say a few words before this slap. He imagines how the scenario might play out and fumes inwardly at Olympia, the woman at the brothel whom Zverkov eagerly anticipated being with that evening. The Underground Man recalls that Olympia once laughed at his (the Underground Man's) looks and refused him.
(Let's just be clear about this: the men left to go to a brothel with which they are all familiar. Olympia is a prostitute who works there, and Zverkov appears to be a regular customer of hers. When the Underground Man once tried to pay her for sex, she said "no thanks.")
The Underground Man continues fantasizing; he imagines that the men, though they will scorn him and kick him out, will finally realize that they themselves are not worth his little finger.
Then he decides that his duel with Zverkov (the result of the slapping) should happen at daybreak. He briefly wonders where he can buy a pistol, especially since he has no money. He decides he can worry about those trivial little details later.
Meanwhile, he urges the driver to make the horse go faster.
He also wonders who can be his "second" (in a duel, each man has a "second," a trusted friend who helps to choose where the duel takes place), especially since he has no friends, but concludes that he could ask the first man he meets on the street to fill the post, and that man would have to agree, out of sheer chivalry.
And then, suddenly, the Underground Man gets shivers down his spine and wonders if, maybe, he should just go home. He curses himself for having coming to this dinner at all.
But then he resolves himself and again urges the driver to go faster.
He continues to run through in his mind every possible scenario. What if they have him arrested? What if Zverkov won't fight? Etc...etc.
He decides that if he is thrown in jail, he'll serve out fifteen years or so, get released, and THEN go find Zverkov in order to magnanimously and publicly forgive him.
At this point, he is basically on the verge of tears, although he knows full well that he had stolen all these scenarios from literature. He feels so ashamed that he makes the driver stop and gets out of the carriage to stand in the snow and ponder.
This lasts, oh, a few seconds before he hurls himself back into the sled and commands the driver onward. He's so eager, in fact, that he punches the driver in the back of the neck. Now that he has resolved himself to slap Zverkov, nothing can stop him.
The Underground Man arrives at the brothel, leaps out of the sled, bangs on the door and is greeted by the madam.
He immediately demands to know where the other men are.
They're not there.
The Underground Man is unspeakably relieved; he feels as though he has been "saved from death."
Then he looks around and focuses on a girl (a prostitute) that has just come into the room. She is young and "grave," and he finds her eyes attractive, though admits she's no beauty. He adds that he would have hated her if she had been smiling.
He looks into the mirror to find that he looks rather atrocious himself; but he's glad of that. He's glad that this girl will find him utterly repulsive.