The speaker of the "Prisoner of Chillon" is a political prisoner. In other words, he hasn't committed any crimes, in the regular sense – he's been imprisoned for his beliefs and the beliefs of his father. In a sense, he's been exiled from the rest of society not only in order to punish him for stubbornly holding to his beliefs, but also in order to keep him from spreading his potentially inflammatory ideals to other people.
The speaker's exile from society is a result of his ideals: the people who imprison him seem to have feared that the speaker's ideas might somehow spread and contaminate others.
By the end of the poem, the speaker's exile is mental and emotional, as well as physical; he prefers his isolation to rejoining society.