check out our:
“Here on this wild outskirt of earth, I shall pitch my tent; for, elsewhere a wanderer, and isolated from human interests, I find here a woman, a man, a child, amongst whom and myself there exist the closest ligaments.” (4.26)
Chillingworth and Hester do have some things in common, after all. Both hold a secret. Both are unhappy. Both have a very desirable skill (Chillingworth is a doctor and Hester is an amazing sewer). And both live on the outskirts of this Puritan society. You'd think they'd have a happier marriage.