Throughout the poem, the speaker focuses very closely on his setting. The irony and wit of the poem, however, comes from the fact that the speaker doesn't focus on any of the things he's supposed to when he's inside the church. Instead, he wonders about practical concerns like whether or not the roof's been cleaned or restored.
Overall, the speaker's reactions to the setting of this poem (the church) symbolize the major theme of this poem, which is the gap between a person who's interested in religious faith and the deeper mystical meaning that only full-blown believers are familiar with. Setting up this relationship between the speaker and the setting shows us that the objects inside the church don't have any inherent meaning, but just the meaning that we give to them. But it also shows that the church does have a vague mystique that keeps the speaker coming back to check out the insides of churches.
The speaker's meditations on what will happen to churches in the future conveys his sense of curiosity, but also his desire to find something he can take seriously about human life. The description of the church, for example, changing from a sacred place to a secular museum—where the "parchment, plate, and pyx [are in] locked cases" (25)—shows a world that the speaker would be more familiar with. Ultimately, Larkin uses the tension between the speaker's ignorance of church symbolism, and his grasp on the physical church itself, as a staging ground for the tension between religious faith and secular skepticism. Eventually the speaker lands somewhere in the middle, in a place of questioning.