The novel itself is a big tangle of different stories, details, and clues—and in that way, it's kind of like a messy ball of yarn. Walter Moody himself kind of seems to make that analogy when he realizes, after hearing all the stories of the twelve men in the Crown smoking room, that trying to sort through all of that is basically a project of unraveling, reflecting:
So I am to be the unraveler, Moody thought. The detective: that is the role I am to play. (I.12.41)
The novel also draws attention to its own tangly ways by having a lot of references to knots. The first moment we get a sense of what Catton is doing with that particular symbol comes pretty early in the book, when Nilssen is muddling through a strange meeting with George Shepard. Even after the meeting, Nilssen is not really sure he gets why Shepard had come there, or whether/with what he was being threatened:
Nilssen sat alone in his office for a long time after Shepard left, turning the gaoler's proposition over and over in his mind. A feeling of doubt was seeding in his breast. He felt that he had missed a connexion somewhere—as if he had come across a knotted handkerchief, balled in the watch-pocket of an old vest, and could not for the life of him recall what the knot was supposed to prompt him to remember—what errand, what responsibility; where he'd been, even, when he tied the corners, and tucked the thing away against his heart. (I.4.234)
Just as the knots in the handkerchief could have a forgotten meaning to the person who finds it, so there was some kind of "knot" in Nilssen's conversation with Shepard that he couldn't unravel. So, in this moment, the knot becomes a symbol of the mysteries and failures of understanding that are flummoxing him.
Also, it's worth mentioning that when we get a flashback to the day Crosbie Wells figured out that Lydia and Frank had stolen all of his gold and important documents, Lydia has Anna tying up ropes into knots as centerpieces for an event that evening (which, not for nothing, is being held to get Raxworthy away from the Godspeed so that Carver can blackmail Lauderback). Given that Lydia's deception is at the heart of that chapter, the heavy focus on knots seems to fit into the book's overall use of knots to draw attention to deceptions, mysteries, and other things whose truths are extremely difficult to untangle.