Unlike adults, young children don't know how to talk yet, so they can't exactly complain about a hot day. That said, they can always start crying if they don't like how they feel. But in any case, Graves is using this opening line to let us know that he's about to draw a contrast between children's experience of the world and adults'.
How dreadful the tall soldiers drumming by. (4)
Just as they aren't able to talk about how hot a day is, children also can't communicate their ideas about how scary a bunch of marching soldiers might look to them. The line might sound repetitive at first, but there's a difference between a hot day and a line of marching soldiers. One is an image of nature, the other is an image of the violence that the adult world is capable of. Children might actually realize how dreadful the soldiers look, but they can't fully understand or make sense of what they're seeing.
But we have speech, to chill the angry day, (5)
Unlike children, we adults are able to talk about our experiences. And talking about experience is a way for us to "master" the things we see, taste, and touch, especially if we can get other people to agree with our experience and reassure us that we're right.
Facing the wide glare of the children's day, (16)
Graves is skeptical about whether adults could ever handle the "wide glare" of children's experience. In saying this, he suggests that the children's world is symbolically wider than the adults, since children aren't able to focus their thoughts in the same way as adults. Adults tend to think of the world in terms of practical information, and language helps them do this. Children, on the other hand, tend to experience everything as a potentially overwhelming experience.
We shall go mad no doubt and die that way. (18)
Here, Graves says that if we adults were ever able to go back and experience the world without any kind of filter, we'd probably go totally insane. Just think about what would happen if every adult in the world suddenly had the mind of a child who couldn't speak yet. The world would be completely overwhelming and we wouldn't know what to do without experience. Worse yet, we wouldn't be able to communicate experience to one another, and the world might more or less fall apart.