Invisible Man isn't heavy on the sex, but when it's there it's impossible to miss. After all, a naked "magnificent blonde" dancing in the middle of a sea of men isn't exactly subtle. Next up, we have Trueblood's incredibly graphic story of incest. Then there's the one scene where the narrator (we can assume) has sex with the woman who was oh-so-fascinated by Brotherhood ideology. It occurs off the page, so to speak, but we "hear" the narrator's thoughts, and they're not exactly pleasant. His rational mind doesn't want to be there, but the strength of his biology overcomes the strength of his ideology. Not exactly the sexiest way to get it on. PLUS, the whole waking up to the woman's husband bit? Highly uncomfortable for the narrator and for us. Lastly, there's Sybil and her rape fantasy, which is never carried out as the narrator plies her with drinks until she passes out.
In sum, sex is not a positive force in Invisible Man, and in fact, it's never depicted as an act of love between two equals who care deeply for each other. This is in keeping with the dominant theme of the novel: invisibility. The magnificent blonde in the first chapter is there purely for entertainment purposes. It's purely her body that matters, and nothing else. Trueblood's story reinforces white perceptions of black culture, and makes the young narrator extremely apprehensive that he, as a young black man, will be identified with Trueblood's act. As for when the narrator has sex, he's not exactly an eager beaver about it. He likens the situation to being a servant called to wash the mistress's back – we might've missed something, but there's no actual sexual desire in that situation. As for Sybil, she really doesn't want to have sex with the narrator specifically – it seems that any young black man would do.