Some afternoons The Giver sent him away without training. Jonas knew, on days when he arrived to find The Giver hunched over, rocking his body slightly back and forth, his face pale, that he would be sent away.
"Go," The Giver would tell him tensely. "I'm in pain today. Come back tomorrow." (13.79-80)
The Giver is in pain because of his awareness. Through the memories he holds, he knows there is suffering in the world—even without the mystical quality of the memories (that is, the way that they physically affect whoever holds them). It makes sense that The Giver would suffer for his knowledge.
"I felt sad today," he had heard his mother say, and they had comforted her.
But now Jonas had experienced real sadness. He had felt grief. He knew there was no quick comfort for emotions like those. (17.10-11)
It looks like everything is relative; Jonas thought he knew sadness and pain before he saw the intensity of the sadness and pain in the memories he received.
"Asher," Jonas said. He was trying to speak carefully, and with kindness, to say exactly what he wanted to say. "You had no way of knowing this. I didn't know it myself until recently. But it is a cruel game. In the past, there have – "
"I said I apologize, Jonas."
Jonas sighed. It was no use. Of course Asher couldn't understand. "I accept your apology, Asher," he said wearily. (17.37-39)
It's fitting that Asher ends the argument with empty words, when what Jonas was trying to do in the first place was get his friend to recognize how hollow his actions and words were.
The Giver shook his head and sighed. "No. And I didn't give her physical pain. But I gave her loneliness. And I gave her loss. I transferred a memory of a child taken from its parents. That was the first one. She appeared stunned at its end." (18.34)
The Giver, because he loved Rosemary, didn't want her to feel physical pain. But his decision to give her emotional pain instead may have been even more destructive.
As he continued to watch, the newchild, no longer crying, moved his arms and legs in a jerking motion. Then he went limp. His head fell to the side, his eyes half open. Then he was still.
With an odd, shocked feeling, Jonas recognized the gestures and posture and expression. They were familiar. He had seen them before. But he couldn't remember where. (19.45-46)
Jonas might not have even recognized death at all had he not seen it happen in a memory. This is how removed every citizen is from suffering and from any comprehension of mortality.
"You suggested, Jonas, that perhaps she wasn't brave enough? I don't know about bravery: what it is, what it means. I do know that I sat here numb with horror. Wretched with helplessness. And I listened as Rosemary told them she would prefer to inject herself.
"Then she did so. I didn't watch. I looked away." (19.55-56)
The Giver, for all his memories, for all the pain he's witnessed, still can't bring himself to watch Rosemary die. This experience is so much more painful for him because he loves Rosemary; the memories, however intense, are still impersonal.