Plums? Icebox? We're totally in a kitchen. In fact, the poem itself could be a note on the refrigerator. It could literally be in a kitchen.
See, "This Is Just To Say" was published in 1934, when refrigerators were first becoming widespread, but were sometimes still called iceboxes, which is what it's called in this poem. Good, thing, too, because the big, clunky word "refrigerator" would stick out in this little poem, don't you think?
It also seems as if our speaker is up early. He's writing notes, instead of sticking around to say something in person. He's got somewhere to be, and he's hungry. He couldn't resist eating these plums, which are probably quite different from his normal morning routine breakfast of Weetabix and a cup o'.
On the other hand, this poem could be set late at night—our speaker has come home when everyone is asleep. He eats the plums as a midnight snack, even though he knew they were probably meant for breakfast. Whoops.
Either way, we're guessing our setting is as simple and everyday as the language of this poem. It's just a normal kitchen, in a family's normal life. Nothing special, nothing dramatic, nothing spectacular. But still, in its own small way, it can inspire poetry.