Hester isn't the only one with a symbol on her chest; Dimmesdale has one, too. In blood.
But we can't quite figure this mark out. It's the physical manifestation of the minister's guilt, but we never quite learn how it gets here. We know that he likes to hang out in a secret room and whip himself, so it's possible that he carved it there himself; but there's also the suggestion that it just sort of arose spontaneously, like maybe it's a guilt-rash, or something. And finally, Pearl comes up with a supernatural explanation: "It is because, when the minister wrote his name in the book, the Black Man set his mark in that place?" (16.32).
However it got there, it shows us that Hester isn't the only one with bodily consequences for sin. She had the baby; he gets the tattoo. Although he refuses to confess and be punished, his sin ultimately marks his body even more permanently than either Hester's scarlet letter or her child: the child grows up, the scarlet letter can be taken off, but Dimmesdale's mark is with him until he dies.