Brother paints himself as Doodle's antagonist. He thinks that every nice thing he did for Doodle, every moment he spent with Doodle, was tainted by his pride and shame. Siblings are famous for antagonizing each other. Older siblings, being bigger, and having more power have been known to take on a bossy, almost parental attitude toward their younger siblings. Because Brother's feels he antagonized Doodle to death the relatively minor things he did to antagonize Doodle are magnified in his mind.
As we discuss in "Character Analysis" Brother is not the sole antagonist. The doctor, Doodle's parents, even the larger community all contribute to Doodle to his death.