The speaker of this poem is an interesting mix of maturity and immaturity. How else can we describe someone who compares a garden where he has to spend a few hours by himself to a "prison," who worries melodramatically that his friends might never return, but who then selflessly tries to spur on a fantastic sunset for the benefit of his buddy Charles. Our speaker is very protective of his friends, but he hates to be apart from them.
One of the speaker's most admirable qualities is his sensitivity to pain and hardship. The poem is addressed to Charles Lamb, and once you learn the back story about the "strange calamity" when Charles's sister killed their mother during a mental breakdown, it's hard to avoid interpreting the speaker's wise, gentle, and reassuring tone except in light of this story. He's like that really nice uncle who wants to take his nephew out to a ball game to take his mind off bad stuff at home or at school.
There are plenty of interesting things going on with the speaker aside from Charles Lamb. For example, plants. The speaker knows a lot about plants and is able to rattle off the names of different trees and shrubs in the area. He would also make a great nature guide with his memory for natural places he has been. He knows exactly where his friends will have to cross the stream across a fallen tree trunk, and he vividly imagines the appearance of the weeds that "nod and drip" on the bank.
One question raised by the poem is whether the speaker has it out for organized religion. When he compares the hills the church steeples and notes that the "Almighty Spirit" of God is hidden within nature, you could be forgiven for thinking that he believes God is nature. More on that topic in the "Calling Card" section.