Our narrator, a grown man, tells us about his memories of Doodle, his younger brother. We jump into the past as the narrator as the narrator begins his story about his brother.
The narrator is six years old when Doodle is born. Doodle is born sickly, and everyone but Aunt Nicey thinks he will die. Daddy even buys Doodle a coffin. When Doodle has lived for two months, Mama and Daddy name him William Armstrong. The narrator doesn't think the name suits him.
The narrator happens to be athletic and actively enjoys the area around his rural home, which isn't far from the ocean. He wishes he had a brother to share it with. Mama tells him Doodle will never be able to run and play like he does. She tearfully suggests that Doodle might not have high brain function either. Ashamed of having a brother like that, the narrator decides to kill the baby by "smothering him with a pillow" (1.5). When Doodle smiles at him he realizes Doodle is smart, and abandons the plan.
When Doodle is two years old he learns to crawl, even though the doctor thinks that his heart condition will keep him from doing even that. He crawls only backwards, reminding the narrator of a doodle-bug. So, the narrator calls his brother Doodle. Doodle soon learns to talk, and begins to show interest in the narrator. Since he can't walk, Brother has to pull Doodle around in a cart. At first the narrator resists Doodle, but then realizes that Doodle is his brother and that he's stuck with him. He takes Doodle to Old Woman Swamp, an extremely beautiful place. Doodle loves it and the two of them spend lots of time there enjoying nature.
Sometimes the narrator is mean to Doodle. Once he takes him to the barn loft and shows him the coffin that was made for him when he was a baby. He makes Doodle touch it under the threat of leaving him there. Even after they leave the loft, Doodle begs his brother, who he calls Brother, never to abandon him.
Doodle is five and he still can't walk. This embarrasses Brother, so he sets out to secretly teach him. Doodle does learn to walk, and on his sixth birthday he shows the rest of the family. Mama and Daddy are very proud. Brother feels bad because he thinks he taught Doodle to walk only out of shame, not because he cares about Doodle. Still, the boys have lots of fun together after that, roaming the area. Doodle begins to tell wonderful stories.
Mama and Daddy decide Doodle will be ready to go to school in less than a year, near his seventh birthday. Brother decides to teach him "to run, to swim, to climb trees, and to fight" (4.1). Doodle is sick most of the winter, and Brother is busy with school. They don't make much headway. That summer, the summer of 1918, Brother decides to teach Doodle to be athletic like he is. Things go well at first, but as Brother presses him harder, Doodle shows signs of weakness and strain. When school is only a few days away, Doodle hasn't learned all the things Brother wants him to.
At lunch on the Saturday before school starts, they find a scarlet ibis, a tropical bird not native to the area, in a tree in their yard. The ibis dies before their eyes and falls from the tree. Doodle buries it, and seems sad.
After lunch Brother takes Doodle to Horsehead Landing and makes Doodle row the rowboat, even though he's exhausted. A storm is approaching. Doodle and Brother are running to get home and out of the rain. Angry and frustrated that Doodle isn't ready for school, Brother runs ahead of him. When his anger calms, he goes back to find Doodle. He finds him curled up under a bush with his head on his knees. Brother touches him and see that he's been "bleeding from the mouth" (4.48). There is blood all over his neck and shirt. He is dead. Brother panics and calls Doodle's name. With tears in his eyes he sees that Doodle looks very much like the scarlet ibis. Brother, crying, puts his body over Doodle's and tries to protect him from the rain.