Okay, so we're going to cheat a bit and bring in some biography here. As soon as we hear about the "summer roses" or the "soldiers drumming by," we know that Graves is positioning us somewhere in England (where he was from) in the years following or even during World War One. Graves published this poem nine years after WWI ended, and there's little doubt that the references to the soldiers walking by is a reference to that war (as opposed to, say, the U.S. Civil War). The reference to the roses no doubt draws on the many rose gardens that exist all over England, and the rose is actually a main symbol of England in general (just check out the national rugby team's logo).
The fact that the roses of this poem have a "cruel scent" (6), however, tends to color everything in this poem with negativity. The dark "overhanging night" also tells us that the general tone of this poem's setting is dreary at best and downright scary at worst. But then again, Graves is talking about how terrible our experience would be if we saw the world in a totally unfiltered way. Cheery fellow, eh? That dreariness, ultimately, comes from inside Graves's head, as he contemplates the pros and cons of language (hint: it's mostly cons). Lucky for us, the poem's setting situates us readers right inside the speaker's head, contemplating the same colossal bummers that he is. Good times.