The idea of comfort is repeated throughout the poem and, like the stars, used as a gauge of what binds and separates the speaker from his wife and son. By the end of the poem, it seems like the only person that's left without comfort is the speaker. Tough luck, buddy.
Line 3: The speaker says that his son's comfort was more important than the stars, which is sort of confusing at first. What do the stars have to do with comfort? But since the speaker already mentioned he wanted to walk outside to "praise the stars", it sounds like he's admitting that his duty to his son was more important than his desire to praise the stars. Um, yeah, that sounds about right.
Line 4-5: Here, the speaker comforts and kisses his son, but to no avail. The speaker says, "my comfort was not enough." Sounds like he can't give his son what his son needs.
Line 15: The speaker, after getting jealous when watching his son breastfeed and finally calm down, asks, "Was my comfort more important than the stars?" Who can blame him for asking a rhetorical question at this point? In some ways, he's starting to sound like the baby of the poem, asking, "hey, what about me? Who's going to comfort me?" Not what you'd expect from a father, exactly, but understandable, right? The man is human.
Line 16-17: Here the speaker finally admits his frustration with his wife and son. He says he wanted to "pull apart / my comfortable wife and son." Take it easy, Dad! Luckily, he doesn't pull the apart, but his frustration does lead him to a moment of clarity by the end of the poem. Geez, who knew trying to love your wife and kid could be so complicated?