The hermit makes a brief appearance in a silly dialogue with the poet at the beginning of the "Brute Neighbors" chapter. Like Thoreau, the hermit lives alone, contemplating Confucian philosophy (only the hermit calls it "Con-fut-see"), trying to seek out spiritual truth. Unfortunately for him, the hermit is unable to get to his profound truth because the poet comes along and distracts him with a fishing trip – a little too easily, actually. It's a moment when Thoreau is clearly poking fun at himself, a nice break in a book that would otherwise be incredibly self-congratulatory.
The hermit also reminds us that complete isolation is not what Thoreau is after. Thoreau seeks solitude, but only so that he can discover important spiritual truths that will be useful to humanity at large.