You might think of Brown as the anti-Jim. He has a lot of the same qualities as Jim: hotheadedness, for one. And he has had many similar life experiences – namely, public disgrace. Yet Brown makes very different choices from Jim (the guy is a pirate after all). His bad choices help to highlight some of the good choices Jim has made, which can be hard to see sometimes, for all his mistakes.
Plus, from Brown's perspective, Jim represents everything he can't or won't become. Jim is a local hero, with a mass of followers and a place to call home. Brown, on the other hand, is a scoundrel and a wanderer, with no place to call his own. No wonder Patusan seems so appealing.