The work of hunters is another thing: (line 5)
The hunters definitely destroy the wall, but their "work" is very different from the work of another force which seems to be in play. Why does the speaker bring the hunters up if he knows that they have nothing to do with the "something" that doesn’t love a wall? What role do the hunters play in this poem?
We keep the wall between us as we go. (line 15)
Just as the speaker and his neighbor have two different perspectives on the importance of and need for a wall, they literally have two different sides. The wall separates their versions of reality.
We have to use a spell to make them balance: ‘Stay where you are until our backs are turned!’ (lines 18-19)
The speaker and his neighbor venture into the realm of (pretend) magic in order to fix their wall. The "spell" they invoke here makes us think of the "elves" that the speaker mentions later on in the poem. The silliness of such things contrasts greatly with the seriousness of the task at hand.