check out our:
Measured by the prisoner's experience, however, it might reckoned a journey of some length; for, haughty as her demeanor was, she perchance underwent an agony from every footstep of those that thronged to see her, as if her heart had been flung in the street for them all to spurn and trample upon. (2.17)
Ouch. Talk about isolation: the entire town has turned out to see Hester paraded through the streets like a criminal. (Well, she is a criminal.) Surrounded by people, she's totally alone.
From the intense consciousness of being the object of severe and universal observation, the wearer of the scarlet letter was at length relieved, by discerning on the outskirts of the crowd a figure which irresistibly took possession of her thoughts. (3.1)
Hester and Dimmesdale aren't the only ones who are isolated: Chillingworth is all alone, too. Fittingly, he shows up on the outskirts of town.
While this passed, Hester Prynne had been standing on her pedestal, still with a fixed gaze toward the stranger; so fixed a gaze, that, at moments of intense absorption, all other objects in the visible world seemed to vanish, leaving only him and her. (3.14)
For Chillingworth, it might as well just be him, Hester, and Dimmesdale. The rest of the town can go to… well, anywhere else, for all he cares. He's got a morality/revenge play to act out.