Where It All Goes Down
London, England in the Early 20th Century
In the beginning of the 20th Century, the city of London was the capital of the largest empire in the world. That being said, we only get a very small glimpse of it (both the city and the empire).
All of the play's action is confined to three places, each located in the very fashionable center of town: Covent Garden, the laboratory of Henry Higgins's apartment at 27A Wimpole Street, and the "drawing room" (think living room) of Mrs. Higgins's apartment on Chelsea embankment. You don't need to know exactly where these places are—just know that they're ritzy, about as far away from the poor parts of London you could get.
The easiest way to understand the setting of the play is to look at who shows up where, and how they're treated. Covent Garden was a large market on London's West End, which in turn was home to many of London's theaters, and it brought together a very diverse crowd of people. Everyone from the rich (like Colonel Pickering), the middle class (the Eynsford-Hills, for instance), to the poorest of the poor—who, like Eliza, were probably trying to make some money off the richer among them. This is the only place we see a bunch of people with (pay attention to this) a bunch of accents mingling freely.
The rest of the action is confined to a lab and a living room, and in both cases the appearance of anyone "lower class" is met with surprise. In the second act, Eliza and her father can't simply walk into Higgins's place: they have to be screened by Mrs. Pearce before they can so much as get through the door.