She called me a filthy name.
I didn't mind that. I didn't mind what she called me, what anybody called me. But this was the room I had to live. It was all I had in the way of a home. In it was everything that was mine, that had any association for me, any past, anything that took the place of a family. [...]
I couldn't stand her in that room any longer. What she called me only reminded me of that. (24.39-41)
Marlowe's rejection of Carmen's (naked!) advances may seem puzzling to some since we know that he complains about his isolation. But Marlowe is definitely the loner type. We can tell from the way he describes his apartment that he has no family and that he considers his bedroom as a sort of safe haven. So maybe Carmen's presence in this room is a threat to his privacy. Sure, he might find her attractive, but not attractive enough to break his solitude.