One of the many problems with being imprisoned in a dungeon is the lack of society. It's easy to lose perspective when you don't have anyone to talk to. In "Prisoner of Chillon," our prisoner doesn't start out alone – he is thrown into prison initially with his two brothers. But after they die, he stops counting the days and just paces around his cell. His mental stagnation is a result of his extreme isolation.
The speaker has to turn away from the view of the island he sees from his grate because the isolation of the island, which is even smaller than the area of his dungeon, cruelly reminds him of the extremity of his isolation.
Because of his extreme isolation, the speaker eventually learns to "love" his imprisonment the way a hermit loves his "hermitage" (lines 374 and 378).