Mr. Summers may act like he's the BMIV (that's Big Man In Village, for all you city-dwellers), but he still has to be sworn in by the mysterious Mr. Graves.
Mr. Graves is never described and he never has a line of dialogue—which, in a short but dialogue-rich story, is like pointing a neon "Symbol!" sign at him.
And Jackson's not going for subtle psychological realism, here: the man's name is Graves, guys. As in the six foot deep hole in the ground. There's a reason he's the ultimate authority in a murderous lottery: his name is where the "winners" of this ritual are going.
Mr. Graves is the postmaster of the village, a position that gives him enormous power since he controls the town's communications with the outside world. Perhaps it's the importance of his work that makes him, literally, the support of the village's tradition. Consider that it is Mr. Graves who brings the three-legged stool to prop up the lottery's black box and it is Mr. Graves who vests Mr. Summers with the right to conduct the lottery in the first place.
Since the only things we really know about Mr. Graves are his name and his job title (and that he's married, a seeming prerequisite for respect in this town), both must be important. It may be the job of postmaster that gives him power in the fictional life of the village, but it's definitely his ominous name that gives him symbolic power as a character in the story.
We think it's significant that Mr. Graves shows up with the stool to hold the black box, swears in Mr. Summers, and then recedes from the story, despite his apparent importance. While the nature of the lottery is left up in the air, all we see is Mr. Summers conducting the lottery like any other small-town event.
But when Mr. Graves does intrude into the narrative, his appearances accompany suspicious hints of the true nature of the lottery.
Remember that, when Mr. Graves comes forward to draw for his family we get an explicit intimation of unease:
[...] all through the crowd there were men holding the small folded papers in their large hands, turning them over and over nervously. (28)
Mr. Graves's draw prompts the narrator to zoom out and survey the fear of the village as a whole, foreshadowing the brutal nature of the lottery; after all, what's at stake in this lottery is the grave, the open secret of the lottery.
Even more ominous is Mr. Graves's appearance in the final paragraphs of the story. Just as we learn that the Hutchinsons have "won" the lottery, suddenly, there's Mr. Graves again, taking little Dave Hutchinson's paper from his clenched fist and opening it.
Dave stares up at him "wonderingly" (69). It may be that this lottery will be Dave's first introduction to mortality, as he participates in the violent death of his mother. It seems only appropriate that it should be Graves who initiates him into the lottery's brutal lessons of life, human nature, and ultimately death.