Symbol Analysis
Like his vacay in hell or his criminal background, the worker is used by the speaker as a kind of symbolic self-description. In this case, however, the speaker's a lot more ambivalent about identifying with the working class. For instance, he says, "I've a horror of all trades. Masters and workers: all peasants, ignoble" (15). So, initially at least, the speaker comes off as a kind of hoity-toity member of the upper class, looking down his snooty nose at the workers of the world.
He later changes his tune, though: "Human labour! It's the explosion that lightens my abyss from time to time" (262). This declaration is a rare (possibly the only) hopeful moment for our speaker. If he's going to feel better, to escape the hell he finds himself trapped in, then it's going to require putting his shoulder to the old grindstone.
But just a few lines later he's flip-flopping again: "Work seems too trivial for my pride:" (267). This kind of inconsistency might frustrate you (we hear ya), but if you've read this far you know that it's just par for the speaker's course. The bigger frustration is that he knows how to cure his ills, but he just can't bring himself to take action to do so. Work and workers, then, represent a way out for this guy. A way that he can see, but not quite get his hands dirty enough to pull off. Instead, he just kicks back and, well, does a whole lot of complaining.