The seven pillars in the dungeon represent the seven members of the speaker's family who have suffered for their beliefs. Pillars are designed to hold something up, so it makes sense that these men, who strive to uphold their beliefs, should be associated with pillars. Of course, the real-life prisoner of Chillon (named François Bonnivard) didn't have the brothers and father described in Byron's poem, but Byron took a lot of creative liberty with the story. Seven is often an important number in fairy tales, so it makes sense that Byron should choose it in romanticizing Bonnivard's story.