Farewell, thou child of my right hand, and joy (1)
While the name "Benjamin" literally means "child of my right hand" (in Hebrew), let's assume we don't know that. For a moment, the phrase "child of my right hand" suggests that the child belongs to the right hand and not to the speaker. This effect is strengthened by the fact that he doesn't say "my child" or "my son" but "thou child." It almost seems like he's talking to a wart or mole on his hand rather than his son. Ew.
My sin was too much hope of thee, loved boy (2)
The same effect as in line 1 is evident here. The speaker calls his son "loved boy," a phrase that sounds just a bit cold. Why not "my adored son" or "apple of my eye" or "shmoopikins"? You know, something that makes the kid seem a bit more like family? The speaker again distances himself from his son, if only just a little bit.
Oh, could I lose all father now! (5)
There is something strange about how the speaker wants to abandon all thoughts of fatherhood. Doesn't he have other children? In addition, it almost seems like the speaker wants to forget the whole tragedy. If he can stop thinking about fatherhood, he can stop thinking about his child too. Weird.