If you've read A Portrait of the Artist as a Young Man, you'll probably remember that Stephen Dedalus grew up in an extremely religious family and that early on he even considered becoming a priest. By the end of that novel, he's broken his ties with the church, and has decided to replace the priestly vocation with the artistic vocation. So in a sense Stephen, one of the main characters of Ulysses, is not religious. But the truth is that he's actually more tormented by religious questions than most of the Dubliners that line up for Church every Sunday. The problem of religion was even more complex in 1904 Dublin because different religions sometimes broke down along political lines: most of the Irish nationalists were Catholics, and most of those who favored union with England were Protestants. Religion, like patriotism, is both an obsession and a danger in the novel; it has enormous value, but how does one escape its narrow-mindedness?
By parodying religious styles in the novel, Joyce reveals that what is often thought of as religious feeling is as much a result of the language in which religious ideas are expressed as it is of the ideas themselves.
Stephen has abandoned the Catholic Church in favor of a personal almost mystical belief in God as present in everything around us. It would seem that the belief might bind him closer to the people around him, but it isolates him because Stephen's religious beliefs are entwined with his guilt over the death of his mother – something with which he has not personally come to terms.