Dimmesdale and Hester both sinned, but she has to wear her shame publicly while he hides behind his ministerial cloak and only confesses just before he dies. She's strong; he's weak. She's isolated; he's beloved by his community.
But you can't hide from God: both of them are marked with a scarlet letter.