The title has a double meaning here. Tricky, we know. At first glance, the title refers to chess, the game Waverly learns to play and master over the course of the story. Chess has a lot of rules, and Waverly has to learn all of them, starting with the formal ones then moving on to the strategic ones and finally getting to the really obscure stuff, like etiquette. So "Rules of the Game" draws a pretty clear connection to the chess skills she picks up.
At the same time, though, there's another game going on between Waverly and her mother. This is a battle for power, a dark game that doesn't come with a rulebook. It still has rules, though, and Waverly has to figure them out if she's ever going to come out ahead of her mother and claim control over her own existence. Chess may come and go, but this game—this battle of the wills—is unrelenting in the story. The title, then, draws a connection between Waverly's chess game and life with her mom. Boom.