LESTER: Both my wife and daughter think I'm this gigantic loser, and they're right.
Yeah, Lester and his family don't have the healthiest of dynamics at the beginning of the film. Or the middle of the film. Or the end of the film. However, Lester at least corrects the loser perception by taking charge of his life, but that brings on other behavior that puts him at odds with Carolyn and Jane.
FRANK: Yeah, yeah—but you said you're partners, so what's your business?
When Frank meets Jim and Jim, it takes him a little while to clue into the fact that they are a couple when they introduce themselves as "partners." Frank is not accepting of that kind of partnership/family, so it takes him a while to even acknowledge that it's in front of him.
FRANK: How come these faggots always have to rub it in your face? How can they be so shameless?
When Frank finally clues in to the reality of Jim and Jim, he goes on a homophobic rant to Ricky. Ricky plays along to make him happy/calm, but he clearly doesn't agree. Later, we learn that Frank's talk is just that—all talk. He's trying to cover up the fact that he has same-sex leanings.
CAROLYN: Lester, I refuse to live like this. This is not a marriage.
LESTER: This hasn't been a marriage for years. But you were happy as long as I kept my mouth shut. Well, guess what? I've changed. And the new me whacks off when he feels horny, because you're obviously not going to help me out in that department.
Lester and Carolyn are exchanging words after Carolyn catches Lester masturbating in bed. She's pretty angry, but Lester isn't feeling apologetic—as far as he's concerned, he's played Carolyn's game for a long time without complaining, and now he's going back to caring more about what he wants. She's not impressed.
LESTER: And your mother seems to prefer that I go through life like a f***ing prisoner while she keeps my dick in a mason jar under the sink.
Things get worse with Carolyn and Lester after Lester quits his job—and now, they aren't even bothering to hide their drama from Jane. At the family dinner where this conversation takes place, Lester shares how castrated he's feeling with his daughter. Needless to say, she doesn't appreciate the confidence.
JANE: Well, you'd better believe I'd hate my dad if he did something like that to me. Wait, I already do hate my dad.
Here, Ricky has been sharing his past family drama with Jane—and if she weren't completely stoned, Jane probably would be horrified (Ricky's dad basically bullied/tortured him into the mental hospital and then left him there for a couple of years). Jane, of course, has her own daddy issues, which is what she's about to get into here.
RICKY: I wish things would have been better for you. Take care of Dad.
After a big blowup with his dad, Ricky is about to leave home. We never did learn what the deal was with Ricky's mom—she is basically catatonic the whole movie—but we suspect she's in some kind of intense submissive state to try to keep her abusive, hotheaded husband calm. Ricky manages to spare a moment to wish/hope for better things for the family he's leaving behind, including his brutish father.