Janice is the big seventh grade bully, the terror of the bus ride and the littler kids. She's big and mean, and at the beginning of the book we don't have much sympathy for the girl. Jess knows how to keep his head down and stay out of Janice's way, but kids as little as May Belle haven't learned that yet. Janice does things like steal little kids' Twinkies and enforce a caste policy on the school bus, where she has a special seat that nobody else can sit in.
It seems admirable and sneaky of Jess and Leslie to write a fake love letter to embarrass Janice when she's mean to May Belle, since they can't beat her up in a physical fight. It's a clever way of dealing with a problem that has to be dealt with: if Jess doesn't try to get back at Janice, he'll lose May Belle's respect.
But they, and we, feel badly for Janice afterwards. We learn later in the book that Janice's bullying comes from a darker place, and is really an example of someone perpetuating a cycle of abuse. In other words, because her father beats her up so badly, the only thing she knows how to do is be mean to other, weaker people. Leslie explains to Jess that Janice's father "really beats her. The kind of beatings they take people to jail for in Arlington" (7.85-87). What gets Janice is not the abuse – as if that's not bad enough – but the fact that her friend blabbed about it and everybody at school knows. That embarrasses her at a totally different level than the silly letter Jess and Leslie wrote.