Kellynch Hall is more than just a really big house: it's the grandest house in the area, because it belongs to the highest-ranked family in the area, the Elliots (even if you don't know anything about how aristocratic rank works in nineteenth century England, the fact that the only other titled person in the neighborhood, Lady Russell, lives in Kellynch Lodge while the Elliots live in Kellynch Hall should clue you in as to who's on top). And rank means more than just who gets to walk into a room first: it means power, and Kellynch Hall is a symbol of that power. According to the old feudal ways, Kellynch is like the mini-White House for its neighborhood, and its owners are supposed to be responsible for the welfare of its surroundings, and to form a moral example for everyone else.
The failure of the elder Elliots to live up to these standards (it's lowly Anne who gets stuck with the duty of "going to almost every house in the parish, as a sort of take-leave" [5.34]) is more than just an individual screw-up: they're failing everyone who's supposed to be able to follow their leadership. And the fact that the Elliots are forced to move out of Kellynch Hall, and the decidedly-not-noble Crofts are able to move in, symbolizes a larger social shift in who's gaining power and who's losing it. Whether this shift is destroying valued traditions (eek!) or bringing in something new and better (yay!) is, however, an open question.