The setting is indicated by the title of the poem: it takes place in a "Spanish cloister." A cloister is the square, open garden at the center of a religious monastery, where, in this poem, a monk named Brother Lawrence likes to grow flowers and fruit trees. The word "cloister" has also come to mean "enclosed," since monastery cloisters are enclosed from the outside world.
The poem's setting does come across as being very claustrophobic. The speaker is trapped in a small community of monks with a man whom – rightly or wrongly – he just can't stand. The speaker's hatred for Brother Lawrence might just be a product of his boredom and restlessness. Perhaps he's dissatisfied with his life as a monk and wishes he had the freedom to be a part of the wider world (this might explain the almost longing detail with which he describes the hair of the women bathing down at the river). Perhaps the speaker feels that religious life is too constricting and confining – too "cloistered."