The poem doesn't shy away from the idea that, because we are born human, we are born already having sinned. This goes back to Genesis in the Old Testament, where the first humans sinned against God by eating some fruit that he explicitly told them not to eat. Hence, the "sin where I begun" reference in the first stanza. The speaker worries that he won't be forgiven for this sin that happened long before he was born. The idea of being already fated to be a sinner is a central concern of the poem.