The hunters definitely destroy the wall, but their "work" is very different from the work of another force which seems to be in play. Why does the speaker bring the hunters up if he knows that they have nothing to do with the "something" that doesn’t love a wall? What role do the hunters play in this poem?
We keep the wall between us as we go. (line 15)
Just as the speaker and his neighbor have two different perspectives on the importance of and need for a wall, they literally have two different sides. The wall separates their versions of reality.
We have to use a spell to make them balance: ‘Stay where you are until our backs are turned!’ (lines 18-19)
The speaker and his neighbor venture into the realm of (pretend) magic in order to fix their wall. The "spell" they invoke here makes us think of the "elves" that the speaker mentions later on in the poem. The silliness of such things contrasts greatly with the seriousness of the task at hand.
Oh, just another kind of out-door game, One on a side. It comes to little more: (lines 21-22)
The speaker reduces the job of mending the wall to an "out-door game," but we think it’s very easy for him to interpret his task in a far more serious way. At this point, we realize how much power our speaker wields in telling the story that he narrates, and in describing the scene that takes place. Do you trust this speaker?
He moves in darkness as it seems to me~ Not of woods only and the shade of trees. (lines 42-43)
Again, he reminds us that the speaker relates this chain of events and this process of mending a wall as he sees it. It could be that the neighbor simply moves in the shade of the trees, but our speaker casts the entire scene in a darker, more ominous light. What kind of darkness do you think this is, and why do you think our speaker makes such a point to distinguish this darkness from the shade around him?