The speaker begins by talking about some mysterious group of people he identifies as "they." At no point in the poem do we find out, specifically, who "they" are. What we do get, though, is a description of what "they" are like. Even this is mysterious, however, because most of the information is contradictory.
We first learn that "they" have the ability to hurt other people, but that they don't do it—but that they really, really look like they are going to do it. Hm. Weird. Next we learn that they have the capacity to "move" other people. We also learn that, even if "they" can move other people, on the inside, "they" remain emotionless and cold.
After all this, we get a big surprise in line 5, when we learn that "they" will "inherit heaven's graces"—and that, according to the speaker, this is just as things should be. Not only that, but he thinks that everybody who isn't one of these cold, mysterious, powerful people is basically a servant. Yikes.
While we're still reeling from this weirdness, Shakespeare completely switches gears in Line 9 and starts talking about the most special flower of summer, and how beautiful it is—even if it does nothing but simply blossom and die. But if this flower happens to become infected, he tells us, it will turn out worse than a weed.
The closing couplet of the poem seems to connect the flower section to the section about the mysterious powerful people. Apparently, by keeping their inner selves hidden, they keep themselves from becoming corrupted, and so are able to remain as beautiful as flowers. The way Shakespeare puts it is much more of a downer, though, and the closing line shows the consequences of corruption, inviting us to smell festering "Lilies" whose stench, he assures us, "is far worse than weeds."