Mr. Summers may act like he's the Big Man of the Village, but he still has to be sworn in by the mysterious Mr. Graves. Mr. Graves is never described, and he never has a line of dialogue, which, in a short but dialogue-rich story, is like pointing a neon sign at him blazing the word "Symbol!" And really, Jackson's not going for subtle psychological realism, here: the man's name is Graves, people. There's a reason he's the ultimate authority in a murderous lottery: his name is where the "winners" of this ritual are going.
Mr. Graves is the postmaster of the village, a position that gives him enormous power, since he controls the town's communications with the outside world. Perhaps it is the importance of his work that makes him, literally, the support of the village's tradition. Consider that it is Mr. Graves who brings the three-legged stool to prop up the lottery's black box, and it is Mr. Graves who vests Mr. Summers with the right to conduct the lottery in the first place.
Since the only things we really know about Mr. Graves are his name and his job title (and that he's married, a seeming prerequisite for respect in this town), both must be important. It may be the job of postmaster that gives him power in the fictional life of the village, but it's definitely his ominous name that gives him symbolic power as a character in the story.
We think it's significant that Mr. Graves shows up with the stool to hold the black box, swears in Mr. Summers, and then recedes from the story, despite his apparent importance. While the nature of the lottery is left up in the air, all we see is Mr. Summers – the symbolic surface of the ritual – conducting the lottery like every other small-town event, like the square dances or teen groups. But when Mr. Graves does intrude into the narrative, his appearances accompany suspicious hints of the true nature of the lottery.
Remember that, when Mr. Graves comes forward to draw for his family (at the prompting of his wife; the position of women in this story is addressed in the "Symbols, Imagery, Allegory") we get an explicit intimation of unease: "all through the crowd there were men holding the small folded papers in their large hands, turning them over and over nervously" (28). Mr. Graves's draw prompts the narrator to zoom out and survey the fear of the village as a whole, foreshadowing the brutal nature of the lottery; after all, what's at stake in this lottery is the grave, the open secret of the lottery.
Even more ominous is Mr. Graves's appearance in the final paragraphs of the story. Just as we learn that the Hutchinsons have "won" a lottery that Tess Hutchinson really doesn't want, suddenly, there is Mr. Graves again, taking little Dave Hutchinson's paper from his clenched fist and opening it. Dave stares up at him "wonderingly" (69). It may be that this lottery will be Dave's first introduction to mortality, as he participates in the violent death of his mother. It seems only appropriate that it should be Graves who initiates him into the lottery's brutal lessons of life, death, and human nature. Or is it human nature? For more on the causality of the lottery, see "Symbols, Imagery, Allegory."