In the 17th century in England, the state religion was Church of England, which had broken off from the Roman Catholic Church about 200 years earlier. The Church of England had gotten rid of or seriously cut back on a lot of Catholic practices like the veneration of saints and emphasis on works (doing good things) rather than grace (having faith that God will save you), but the Puritans thought they hadn't gone nearly far enough.
Puritans were a sect of Protestant Christians influenced by Calvinism, the idea that salvation is predestined. You'd better hope you're one of the lucky ones, because there's not one thing you can do to get yourself saved, not even going to church all the time. How do you know if you're saved? Well, you really like going to church. Religious behavior (and worldly success) was a result of salvation, but also proof of it. Because Puritan communities were really into purity (duh) sins were rooted out and punished harshly.
Unfortunately for them, Church of England was basically compulsory. You had to show up at least a couple of times a year, and you could be seriously persecuted for practicing different beliefs. So, what's a Puritan to do? Obviously, set sail on a treacherous, many-weeks-long journey to settle in a (to them) barbarous, wild land—basically like going to live on the moon—so you can root out sin in peace.
This is the setting of The Scarlet Letter. It's a community specifically designed to be religiously pure, which means being secularly strict. In the first chapter, we get a mini tour of the most important town buildings and structures: the prison and the town scaffold. Law and religion form the heart of the town.
But it's not all heavy oaken doors and iron studs. No matter how many prisons they build, the Massachusetts Bay Colony is surrounded by vast expanses of forest and ocean. The colony is like an island in the midst of wilderness, and the sense of the unknown and unexplored is tangible.
Nature (as represented by this ocean and this wilderness) is far larger than civilization (as represented by the town itself). The colonists are on the frontier, having left the Old World of England in exchange for the New. And there's always the sense that, while inside the town is all order and law, right outside is the unknown: witchcraft, Indian medicine, adulterous relationships, and—just possibly—some new and better ways to live.