The title has different levels of meaning. On the most basic level, the poem is a "rhyme" – that is, it has rhyming verses – told by an old sailor, or mariner. Simple enough.
But why is "rhyme" spelled "rime"? Ah, now it gets interesting. In addition to "rhyme," the word "rime" means frost, and specifically the frost that forms in fog and wind when the temperature cools down. "Rime" often forms on the windy side of sails and ships. Much of the poem takes place in the Antarctic, in a "land of ice and snow," and you expect to encounter a lot of rime in that climate. Furthermore, the Mariner himself is described as being "frosty" in some respects. For example, his beard is described as frosty or, "hoary" (7.142). If you wanted to turn this idea into symbolism, you might say that the Mariner's soul is covered with a layer of frost until he learns to have pity on his fellow creatures.
Finally, he's an "ancient" mariner because, clearly, he's very old. "Ancient" makes him sound like some timeless artifact, one that has always existed and always will exist.