check out our:
From the intense consciousness of being the object of severe and universal observation, the wearer of the scarlet letter was at length relieved, by discerning on the outskirts of the crowd a figure which irresistibly took possession of her thoughts. (3.1)
Hester and Dimmesdale aren't the only ones who are isolated: Chillingworth is all alone, too. Fittingly, he shows up on the outskirts of town.
While this passed, Hester Prynne had been standing on her pedestal, still with a fixed gaze toward the stranger; so fixed a gaze, that, at moments of intense absorption, all other objects in the visible world seemed to vanish, leaving only him and her. (3.14)
For Chillingworth, it might as well just be him, Hester, and Dimmesdale. The rest of the town can go to… well, anywhere else, for all he cares. He's got a morality/revenge play to act out.
Dreadful as it was, she was conscious of a shelter in the presence of these thousands of witnesses. It was better to stand thus, with so many betwixt him and her, than to greet him, face to face, they two alone. (3.14)
Sometimes there's safety in numbers. Chillingworth can't exactly confront her while she's standing up on stage in front of "thousands" of townspeople—not if he wants to carry out his insane revenge plan, anyway.