The speaker of "The Prisoner of Chillon" is imprisoned, originally, for refusing to back down. Yes, this could look like stubbornness, but it's a trait we're meant to admire: the speaker's father and brothers all die for their beliefs, and the speaker and his two remaining brothers are thrown into a dungeon for holding to the same cause. We're never told what the beliefs actually are – you can insert your own favorite cause or political belief, if you like. The point isn't the actual cause the speaker's family was fighting for; the point is that they believed in something strongly enough to die for it.
Although we know that the historical prisoner of Chillon, François Bonnivard, was imprisoned for taking a stand against the ruling Duke of Savoy, Byron leaves the details of his speaker's beliefs a blank in order to give the poem a more universal appeal.
"The Prisoner of Chillon" is not about a particular political stance; by leaving the prisoner nameless and his beliefs unspecified, Byron makes the poem about perseverance and principled resolve more generally.