In the opening line of the poem, Graves points out that children—unlike adults—aren't able to talk; they're not able to put their experiences into words. As we find out later in this poem, Graves takes this to mean that experience strikes children with a lot more emotional force than it does adults.
How hot the scent of the summer rose, (2)
In line 2, Graves continues to talk about how children aren't able to take what comes in through their senses and arrange their experience into something they can communicate to another person. In this case, we might think the smell of a rose is something pleasant. But for a child who doesn't really know what to do with their experience, the smell of a rose might be awful.
But we have speech, to chill the angry day, (5)
Unlike kids, we adults have speech and are able to talk about our experiences, sometimes in really descriptive ways (like Graves does with his poetry). And apparently, having the ability to speak helps us to "chill the angry day." We don't know why yet, but Graves claims that our ability to speak manages to make our harsh experiences less harsh.
And speech, to dull the rose's cruel scent. (6)
Once again, Graves tells us that our ability to speak and to turn our experience into something say-able manages to make our experience easier to bear. But the fact that he uses the word "dull" here already suggests that there might be a downside to making experience easier to handle. We might make our unpleasant experiences easier to bear, but we might also end up dulling our more joyful experiences, too.
We spell away the overhanging night, (7)
For Graves, the act of spelling out a word is basically the same thing as casting a spell. And in this case, you can also read "spell" as being a shortened form of "dispel," meaning to make something go away or dissolve. So in this sense, language has a sort of supernatural ability to dispel the dark night that might scare a young child. As adults, we can compare the dark night to hundreds of other dark nights we've experienced, and thus make it seem less scary.
Facing the rose, the dark sky, and the drums, We shall go mad no doubt and die that way. (17-18)
By the end of the poem, Graves is telling us that even though language tends to dull our experience and our souls, we still can't live without it. We can't just go from being rational creatures to having no language to give shape to our experiences. The shock would be just too much, and we'd be totally overwhelmed by all the information our brains usually filter out.