Before we get started, let's just go ahead and define an "angle of repose" for all you non-engineers out there: it's the angle at which a material will stop sliding.
In other words, if you pour a bucket of sand on the ground, its angle of repose would be the way that it settles onto the ground. Make sense? No? Check out this video if not.
So, what does this have to do with anything? Well, Lyman makes our job easy because he lays it right out for us: he wants to figure out how his grandparents "clung together [...] rolling downhill into their future until they reached the angle of repose where I knew them" (4.1.54). In other words, he wants to figure out how they ended up the way they ended up because whatever they did kind of worked.
While Lyman's understanding of his grandparents' angle of repose (and his own) changes over the course of the novel, he remains obsessed with the concept at all times.