Conan Doyle is writing his Holmes novels and stories at a time when money is pouring into the U.K. from its colonial territories. But with rising capital comes huge anxiety over the social effects of greed, theft, and instability. The Blue Carbuncle and the Beryl Coronet are probably the most obvious examples of riches coming in from outside and then tempting good (or not so good) men and women to commit evil.
But money isn't the only thing that's going into and out of England at an amazing rate: there are also huge movements of people, both to the colonies and back again. And when they come back, they're not always the better for having been away. Consider Dr. Roylott's extreme violence upon his return from India, where he already showed himself capable of murder. And how about the torment Charles McCarthy brings to John Turner, as a remnant of his wild past in Australia? Even the cruel Hosmer Angel/Mr. Windibank trick played on Mary Sutherland depends on her stepfather's business trips to France. Increased contact with the foreign can be a good thing, as with Holmes's encounter with Irene Adler, straight from Warsaw. But it also can provide increased opportunity for evil to take advantage of vulnerable people.
In Sherlock Holmes stories, the British colonies seem like centers of lawlessness waiting to introduce their legacies of crime into England itself.
Conan Doyle makes an analogy between people of color and dangerous places, as in his racist descriptions of the opium dens of "The Man With the Twisted Lip."