The poem argues that desire, both of the sexual and non-sexual varieties, has its roots in memory. We desire things which remind us of other things that we have lost, but want to recover. We are "thirsty" for the past. The poet remembers making love to a woman who reminds him of his favorite "childhood river," and he even comes to the surprising conclusion that this mental connection is more important to him than any of the woman’s specific traits. But – here’s a real mind-bender – maybe the pastoral setting of the poem is beautiful because it reminds him of the woman, who reminds him of his childhood. Desire might just be a trip further and further back into the distances of memory.
The speaker thinks that the natural scenery at Lagunitas is beautiful because it reminds him of his encounter with a lover.
If desire is a longing for the past, then it can never be fulfilled.