The unnamed "prisoner of Chillon" is alone in a cell by the banks of Lake Geneva, in Switzerland, where he has grown old as a prisoner. He says that his father was executed for his beliefs, and all six of his sons have suffered persecution for the same reason. Three of the six sons died outside of the prison: one was burnt at the stake and two died in battle.
Our narrator, the prisoner of Chillon, was originally imprisoned with his two remaining brothers. He was the oldest of the three, so he tried to keep their spirits up, even though the three of them were chained to individual pillars in a large cell and couldn't even walk around. The middle brother, who was an outdoorsy, huntsman type of guy, just couldn't bear to be imprisoned, so he gave up hope (and food) and died. Our prisoner was left with the youngest brother, who was cheerful and patient. But, unfortunately, he also wasted away and died.
The prisoner almost gives in to grief, but is revived when he hears the singing of a bird outside his window. It reminds him that there's beauty and hope in the world. So he clings to that thought and survives. Years later (the prisoner stopped counting the days ages ago), the guards arrive to set him free. But he's been in jail so long that he doesn't know what to do with freedom once he has it. Everyone he loves is dead, and he has nowhere else to go.