In a sense, Lord Jim is all memories. Marlow tells Jim's story through his own memory, and the memory of other sources (including Jim himself) from whom he learns about Jim's life and death. For many of these characters, Jim in particular, the past and memories of that past are inescapable. Jim's fateful choice aboard the Patna sticks with him for the rest of his life. Memories of it consume him, shaping his choices and his chances. Marlow seems keen on telling us this story to show us just how powerful these memories can be.
Jim allows his memories to haunt him. It's a choice, and it's one he didn't have to make.
Jim represses a lot of his memories, which is probably why they bother him so much. It's nothing a little trip to Freud wouldn't cure.