At the end of the day, this is Jude's story. His dreams, desires, flaws, and actions drive a major portion of the novel. The thing that makes Jude so intriguing as a protagonist is that he is easily shaken off course, whether it be by love, lust, confusion, or some outside force. Jude's desire to go to Christminster to become a scholar or a religious man is what moves things along early on.
However, that desire to get to Christminster ends up being a plot device that serves the story of Jude and Sue. They meet in Christminster when Jude finally makes it there, and his decision to bring Sue and their family back to Christminster is what ultimately destroys everything. Almost instantly upon arrival, Little Father Time proclaims, 'I don't think I like Christminster!' (6.1.29). If Jude could have ever let the Christminster dream fade, there is a chance he could have been happy in his life.