The hermit only appears one time in the poem, at the end of the first stanza. A hermit is a person who secludes himself from the world and lives alone, usually for religious reasons. We don't even know whether there is really a hermit living in the woods near the Wye. The speaker is just musing about the possible source of the smoke he sees rising from the trees. Maybe the speaker thinks of a Hermit because he'd like to retire into the woods himself and live in seclusion from the rest of the world to commune with nature.