This theme doesn't come up nearly as often as love and death, but it's a really neat and important part of "Annabel Lee." This isn't a long poem, but Poe manages to weave all kinds of different themes into it. In this case he gives us just a hint that Annabel's family doesn't think much of him, and wants to tear the young lovers apart. In a sense, family gives him a way of talking about the pressure of outside society, all the people who can't understand how pure and true his love is. This is definitely an "us against the world," Romeo and Juliet kind of poem.
The presence and the actions of Annabel's family help to show the distance between the speaker's attitude and those of the larger society.
The family, like the angels and the demons, is the symbol of a repressive, alienating, evil world, in which the only beautiful thing is the love of Annabel Lee.