Guilt is pretty much one of the worst feelings, and our main squeeze really struggles with it. Some of the guilt he experiences is legitimate (like when he hurts Rufus's feelings) but some is self-imposed guilt over things that Kenny really can't control (like what happened in the church). Meanwhile, Byron seems totally guilt-free—until the death-by-cookie incident. The lesson we learn from The Watsons Go to Birmingham—1963 is that feeling guilty isn't always related to being at fault.
Kenny let Joey down in the church; he should have stayed until he was sure she was okay.
Kenny blames Byron for the fact that they don't have a good relationship, but it's partly Kenny's fault, too.