In addition to that kind of chilled-out sad feeling the speaker has, there's another kind of bad feeling in this poem. This one is a lot less fun. Here we're talking about real suffering. Well, not like being tortured, or going to the DMV, but just the everyday pain of living. You know this feeling? When getting through the day feels like a drag, when all the things you have to deal with seem to be piling up on top of you? Well, we hope you don't feel that way all the time, but it's a big part of what "The Day is Done" is about.
The difference between sadness and suffering is at the core of this poem, and the whole thing is an effort to give a precise definition of those mysterious emotional states.
While the speaker insists at the beginning that he is not in pain, his descriptions of his feelings become gradually more intense and, by the end of the poem, it is clear that he is really suffering.