check out our:
“Here on this wild outskirt of earth, I shall pitch my tent; for, elsewhere a wanderer, and isolated from human interests, I find here a woman, a man, a child, amongst whom and myself there exist the closest ligaments.” (4.26)
Chillingworth and Hester do have some things in common, after all. Both hold a secret. Both are unhappy. Both have a very desirable skill (Chillingworth is a doctor and Hester is an amazing sewer). And both live on the outskirts of this Puritan society. You'd think they'd have a happier marriage.
On the outskirts of town, within the verge of the peninsula, but not in close vicinity to any other habitation, there was a small thatched cottage. It had been built by an earlier settler, and abandoned because the soil about it was too sterile for cultivation, while its comparative remoteness put it out of the sphere of that social activity which already marked the habits of the emigrants. It stood on the shore, looking across a basin of the sea at the forest-covered hills toward the west. (5.4)
Well, obviously Hester lives on the "outskirts" of town. (Hawthorne must love that word.) She's an outcast; she can't exactly pop over the neighbors every time she wants to borrow a cup of sugar or gossip about that adulteress at the end of the block.
In this manner, Hester Prynne came to have a part to perform in the world. With her native energy of character and rare capacity, it could not entirely cast her off, although it had set a mark upon her more intolerable to a woman's heart than that which branded the brow of Cain. (5.8)
Hester may be an outcast, but she's not entirely isolated. She manages to win a place for herself by sheer hard work—although we're not 100% sure why she even bothers. Why would you want to be a part of the community that had cast you out?