Our speaker takes great pains to describe the setting of this New England countryside. He tells us right off the bat, "Something there is that doesn’t love a wall/ That sends the frozen-ground-swell under it,/And spills the upper boulders in the sun," (lines 1-3). In doing so, he points a big, fat finger toward nature. Nature seems to be the unnamed culprit who, in addition to hunters, continues to destroy the wall. As the poem unfolds, we learn how spring (and all of its feverish weather and spirit of new life) makes our speaker a bit mischievous. We see in this poem the sharp contrast between the natural and the artificial, nature and man.
The "something" to which the speaker refers is nature itself.
The neighbor does not trust nature to bring peace.