You could think of this poem as an invitation to the dinner of this couple. And dinner is a casual affair, remember? Maybe that's why this poem doesn't have any organized meter.
That's not to say that "The Bean Eaters" doesn't have any rhymes – it does. In fact, it's got enough of a rhyme scheme to make it seem as though it's formalized. (In case you're wondering, it's actually AABA BCDC EFDF.) Funnily enough, there is a form in which the second and fourth lines of a poem rhyme – it's called the "common meter." Common. Sort of like casual. Get it?
Unlike common meter, however, this poem doesn't have a regular metrical pattern. It's not nearly as regular or formal as even common meter tends to be. That's probably because it describes people who aren't too concerned with preserving a formality in their lives. It's enough for them to get food on the table – why worry about speaking (or writing) in iambic tetrameter?