It's been said that everybody dies alone, and this line hammers the fact home. Not only is he alone in death, but he's apparently cursed with the ability to keep speaking, even though no one can hear him. That's gotta be torture.
I was much further out than you thought (3)
If he's swimming, then he's too far from shore or the other swimmers. We have to wonder why he's so far out. Was he being reckless? Did he overestimate his swimming ability or mistake the water's depth? Maybe a kiddie pool would have been more his speed.
Oh, no no no, it was too cold always (9)
The phrase "out in the cold" can mean "excluded." So being out where the water is too cold seems like a metaphor for being cut off from the warmth of home and companions. The dead man felt he was always excluded from these basic comforts.
I was much too far out all my life (11)
Apparently this is about more than just swimming, unless he was trying to set a Guinness World Record. The isolation described is life-long, so swimming is probably a metaphor for living. Do you feel like the dead man is more or less to blame for the isolation when you learn that he's been alone for so long? Are there reasons for the isolation that don't put blame on anyone?