So many of Chaucer's pilgrims are not what they appear or what, in a perfect world, they would be. The Pardoner pretends to be a holy man intent on saving souls with his relics; in reality he's a money-grubbing liar. The Friar should be living a life of poverty and chastity; instead he's living the high life with rich people and whores. The darkness that is oppressing the pilgrims is no one person or figure, but rather a general state of sinfulness.
The pilgrims' pride and sinfulness brings them into conflict with one another as the pilgrimage progresses. The 'felaweshipe' that is the goal of any group of pilgrims eludes these ones despite the Host's and others' best efforts to ensure it, because these pilgrims' sinfulness puts them into a state of division.
Since the pilgrims never arrive at Canterbury, which is where their sins would be cleansed, they never reach a state of joyful union. The whole point of a pilgrimage is to receive absolution for your sins and thus, perfect union with God. This perfect union with God would also lead to perfect union with your fellow Christians. The pilgrims would at last be able to achieve perfect fellowship. But since we never get to see them reach Canterbury, we never get to see them in a state of joyful union.