It's clear to everyone in "Not Waving but Drowning" that the dead man suffered, but just how long or why is a matter of debate (or disinterest). As readers we initially think he's moaning about his death. Likewise, the people in stanza 2 think the saddest thing was the suffering he experienced at the moment of death. But the dead man himself tells us it was much worse than we suspect: it wasn't his death that was unbearably painful, it was his entire life.
It's simple. The dead man suffered because he kept himself away from other people.
The other people should have seen his suffering and come to his aid, however he pretended to feel. It was their coldness that killed him.