That night in bed, Alvy tries to initiate sex with Annie, but she ain't having it. He says she used to be interested in him, big time. Annie says it's natural to cool off after a while, and that Alvy was married before, so he should get that.
Time for another flashback. This time, it's to Alvy's first marriage. What's up, Allison?
We catch Alvy and Allison's first meeting, backstage at a political rally where she was working and he was performing.
He reduces her to a cultural stereotype based on her politics, appearance, and graduate thesis title, but she's still smitten for some reason.
Cut to Alvy and Allison in their bedroom arguing about the JFK assassination. He thinks there's a massive, Oliver Stone-level conspiracy. She thinks he's going on and on about a conspiracy to avoid sleeping with her.
Alvy breaks the fourth wall to say Allison's right, and he doesn't know why he sabotaged their relationship. She was a cool chick. Maybe it has something to do with that old Groucho Marx joke.