This play is chock full of baby images. It seems like Albee slips one in at least every other page. First of all, George and Martha call each other "baby" all throughout the play. Martha also calls Nick that a few times too. One of those occasions is the first time they kiss. Nick touches her breast and she pushes him away saying, "Take it easy, boy. Down, baby." (7.31)
George and Martha also often refer to Nick and Honey as if they were children. When the young couple first arrives, George greets them with, "You must be our little guests" (1.64), while Martha says to them "Hey, kids…sit down" (1.175). Later on George calls them "tots" (3.232) which is an even more blatant baby reference.
Another baby image is when George describes Honey on the bathroom floor, saying she's "Peaceful…so peaceful […] sucking her thumb […] rolled up like a fetus" (2.750-2.752). We should also point out that Martha sometimes talks to George in baby talk, usually by begging for a drink.
The reason for all this baby imagery becomes pretty clear when we learn that both couples have had imaginary children. Nick married Honey because she had a hysterical pregnancy. She swelled up as if she were pregnant, but it turns out it was all in her mind. And, of course, there is George and Martha's imaginary son, whose "death" marks the climax of the play.
Both couples seem to be fixated on the fact that they don't have children. One of Honey's few sincere moments is when she plaintively cries, "I want a child. I want a baby" (3.362). Then, when George "kills" he and Martha's son you'd think that a nuclear bomb went off. Martha is completely devastated. Both couples' fixation on children could represent any number of things. Perhaps, they wish to fill the holes in their relationships with a child. Maybe, they feel a child would project the image of a happy family. Possibly, they hope that a child would bring real meaning to their lives. Any or all of these things could be true.