Until he comes home to Brideshead Castle to die, most of what Charles (and we) knows about Lord Marchmain is second hand. It starts with Anthony early in the novel, who tells the story of Marchmain’s affair and his current exile from his wife and in fact all of England. Interestingly, Anthony takes Lord Marchmain’s side and condemns Lady Marchmain.
Next we know, Sebastian explains that he’s the only of his siblings maintaining any sort of real relationship with his father. Sebastian, too, sympathizes with his father, so for the second time Charles is inclined to side with Lord Marchmain and against Lady Marchmain. When Charles finally gets to meet the man, we once again get more information from a second-hand source than we do from our first-person narrator's observations. This time, it’s Cara.
Cara’s long description of Lord Marchmain’s rocky relationship with his wife is there for a reason: it’s one more step in Charles’s education regarding love. It also sets up a parallel for us between Lord Marchmain’s marriage to Lady Marchmain and Charles’s "romantic friendship" with Sebastian. Both are first loves, as Cara says. Both are immature, and both act as forerunners to a later, more mature relationship. In the case of Lord Marchmain, this second love is presumably what he has with Cara.
Lord Marchmain is also a general parallel for Sebastian and his bad habits, as Lady Marchmain points out in the following passage:
"You see, it's all happened before. […] I mean years ago. I've been through it all before with someone else whom I loved. Well, you must know what I mean – with his father. He used to be drunk in just that way. […] But the running away – he ran away, too, you know. It was as you said just now, he was ashamed of being unhappy. Both of them unhappy, ashamed and running away. It's too pitiful."
Of course, Sebastian’s character and decisions are very, very different from his father. He chooses suffering to be closer to God while Lord Marchmain essentially renounces his faith by moving to Italy and living with his mistress. That Lady Marchmain can’t recognize these differences only suggests how little she understands her son.
Speaking of faith, let’s look at what happens when Lord Marchmain comes back into the picture at the end of Book Two. The first thing that Charles notices is that, actually, this guy is new kind of a jerk. "I had always been aware of a frame of malevolence under his urbanity," he says, "now it protruded like his own sharp bones through his skin." But it doesn’t take long to figure out that Lord Marchmain’s new personality is essentially just a reaction to his illness. He really, really doesn’t want to die.
Now let’s check out that death scene. By this point, Charles has tirelessly argued with Bridey and Cordelia and even Julia against having a priest come to perform the Last Sacrament on Lord Marchmain. "No one could have made it clearer," Charles says, "all his life, what he thought of religion. They’ll come now, when his mind’s wandering and he hasn’t the strength to resist." Charles gets pretty worked up over the issue, and it is in fact the final dividing factor that destroys he and Julia’s relationship. Charles comments that "the fate of more souls than one [is] at issue," and indeed this turns out to be true. Lord Marchmain decides not only if he will or will not see a priest, but also the heir to Brideshead and, in a way, whether or not Charles and Julia will stay together.
What does Lord Marchmain have to do with Charles and Julia’s relationship? Plenty. When he – who all his life has opposed religion – manages to make the sign of his cross and willingly receive the Last Sacrament on his death bed, he brings Julia back to God. It is right after this touching scene that Julia decides she needs to accept God’s love and break it off with Charles. She was inspired by her father’s actions.
The other issue at hand is Charles’s own conversion to Catholicism, which we talk about in "What’s Up With the Ending?" The first time we see a shift in Charles’s attitude toward Catholicism moments before Lord Marchmain’s death, when they wait for him to accept or refuse the Last Sacrament. "I suddenly felt the longing for a sign," explains Charles, and after he sees it suddenly remembers a scene from the Bible. These are the first stirrings of what will ultimately lead to his position as a Catholic by the time he’s in the army in the 1940s.