Nobody is forced to leave town in "To Helen," but the speaker nevertheless describes an experience that sure sounds like exile. He compares himself to a "weary way-worn wanderer" and points out that he's done a lot of roaming. Helen's beauty is the key to his return. Hmm. It sounds like exile, smells like exile, but in this poem it's more of the metaphorical variety. In other words, the speaker realizes that, before he met Helen, he felt alone, banished, excluded—in short, like an exile.
The speaker feels like an exile before he meets Helen. This implies that life itself is a form of exile, until we meet somebody we love, find our other half, make a love match, get a hit on our eHarmony page, etc.
Exile is about feelings more than anything else. The speaker isn't a literal exile, but his feelings of loneliness and desperation certainly resemble those of someone who is looking to get back to the comfort and safety of home.