We really only know two things about Zdena, Mirek's former lover. First, she's not exactly a Betty (translation: she's super ugly). It's so bad that Mirek thinks about her lack of beauty and feels that "Zdena's big nose cast a shadow on his life" (I.8.5)—he believes their history together has made it harder for him to hook up with true babes.
There's one other detail you should know about Zdena: she was "loyal to the garden where nightingales sing" (I.8.2). In other words, she's a supporter of the Communist regime. That immediately puts her at odds with Mirek, who is a known dissenter.
Mirek wants to get his love letters back from Zdena. Not exactly because he's embarrassed to have loved her—he really just wants to control all the pieces of his life story. He knows the regime will take him out soon, and he wants to make sure that he gets to shape the story that's going to be spread around afterward.
Even his old feelings for Zdena can't shake the intensity of that feeling. When she won't give back the letters, Mirek feels all the existential rage: "That a piece of his life remained in Zdena's hands was unbearable, and he longed to hit her over the head with the big glass ashtray on the coffee table between them and take away his letters" (I.14.8).
Is it simply because Zdena won't do what Mirek wants? Or is there something else still simmering beneath the surface? Zdena doesn't care—and Mirek's not telling.