Surprise! It's Roger Chillingworth. (What, you weren't surprised?) When he finds out that Hester has committed adultery, he seeks revenge by tormenting her lover psychologically, so you could argue that Chillingworth is the primary roadblock preventing Hester and Dimmesdale's happiness. On the other hand, Dimmesdale's conscience seems to have a bit to do with it as well.