Frederick Douglass doesn't start out with much of a family. His father is probably his white master, or some other white rapist. He barely knows his mother: he only remembers meeting her a handful of times before she died, and he wasn't even allowed to go to her funeral. So when he talks about family, it's as something that slavery prevented him from ever having. Almost the only time he even mentions his cousins and grandmother, for example, is when he's telling the story about how his grandmother watched all her children and grandchildren get sold away from her, never to be seen again. And though he doesn't say much about his own marriage, he can only get married (and start his own family) after he escapes to freedom.
One of the greatest tragedies of slavery is that the slave has no family, because this loss can never be fixed. Even after Douglass becomes free, he still has no mother or father.