After his Patna disgrace, Jim refuses to have contact with people he knew before, including his own family. Though we feel sympathy for his unfortunate situation, Jim is the author of his own exile. This sends our boy on a long search for a new place to call home, which you might say he finds on Patusan. Lord Jim traces the great deal of lonely wandering before he manages to carve out a little corner of the world for himself. Though he does manage to make a name for himself in Patusan, Jim's self-imposed exile is always in the back of his and our minds.
Conrad argues that there is no place on earth for Jim to escape the Patna incident as long as he refuses to let go of his past, so all this exile business is totally pointless.
Jim isn't the only one in exile. Most of the other characters are far from home, including Marlow, who copes with his exile by devoting himself to storytelling.