The speaker's sense of place, family, and identity are all wrapped up in sensory memories of the natural world. He remembers the textures, smells, sounds, and, of course, sights of growing up with his father and grandfather working in the potato fields. In "Digging," the speaker looks back with fondness on his early encounters with the natural world. He feels rooted in the earth of the potato fields and peat bogs. Although in his adult life as a writer he has in some ways moved out of the natural world, he still writes about it, and it doesn't seem like he's going to forget nature any time soon.
Although the speaker is a writer, and isn't necessarily always outside in the natural world, his work will have all kinds of natural things in it, because he is still affected by the memories of his childhood in nature.
The speaker feels guilty that, as a writer, he has strayed so far from his roots in the natural world.