We have to tread gingerly here. We can assume we've got a speaker who's checking out some trees and who starts to think of all sorts of things. (By the way, Ashbery often tried to capture the goings-on in our minds, hence the weird poems we get.)
The trees start to remind him of a relationship he has, the sexual orientation of which is unclear. The poem goes on to detail connections between the trees, the speaker's relationship, and the experience the speaker has with both of these things. The speaker doesn't try to make "sense" of the experience though. Instead, he wants to experience everything for what it is, rather than dousing it in something run-of-the mill, like meaning.