Babylon is a term taken from The Book of Revelation in the Bible. The Biblical Babylon is a city characterized by extravagance, debauchery, and sin, but it is really taken as a symbol rather than an actual place. (In a famous passage, Babylon is actually represented as a woman, the so-called "Whore of Babylon.") The consequence is that we can understand "Babylon" on several levels in the context of Fitzgerald's story.
To interpret Babylon as a literal city means that we're looking at Paris as the place of sin and indulgence. Charlie is revisiting Paris, so in this sense he is returning to revisit "Babylon." But Charlie isn't just revisiting a place; he's revisiting an entire lifestyle that he's left behind, and entire state of mind to which he prescribed for a decade. "Babylon" isn't just Paris; it is Charlie's former life.
We should note that the Bible foretells of the destruction of Babylon. According to the story, Babylon will be destroyed on account of its sins. Charlie is living through the fall or destruction of the world he once knew, as he pays the price for the extravagance of his former lifestyle. Check out his descriptions of Paris and the way it has changed.
Fitzgerald's title sets "Babylon Revisited" against a religious or mythological backdrop, which highlights other religious connections we might otherwise miss in the text. One interpretation of Charlie's attempts at atonement is that he is in purgatory. Purgatory is a Christian term for the place where souls go before heaven. In purgatory, one pays for one's sins and is purified. When Charlie asks how much longer he has to pay, he effectively wonders how long he'll have to wait in limbo before he finishes atoning for his former sins.
There's also an interesting passage in which Charlie sees that "the two great mouths of the Café of Heaven and the Café of Hell still yawned – even devoured […] the meager contents of a tourist bus" (1.55). And we can't forget the sinister side of Duncan and Lorraine, who pursue Charlie like furies through Paris. We see that, Fitzgerald complicates and deepens our reading of the text in his choosing of this title.