The landlords pop up frequently as those nasty brutes harassing their tenants for money. In fact, the narrator mentions "teaching landlords to have at least one degree of mercy toward their tenants" as a failed solution for poverty-stricken Ireland (28).
Basically, these guys are bad news. But who are they? Although Swift is certainly frustrated at England, he's referring to people who reside in Ireland. Robert Phiddian suggests that the landlords are the Anglo-Irish elite, or the wealthy folk who don't care about their country (source). Swift may have been pretty upset at the British, but he chose to direct his critique where it would hit home.
The narrator certainly isn't letting the landlords off the hook. If they won't accept late rent, maybe they'll shell out some dough for dinner. Really, the landlords could be seen as anyone who looked the other way when the country was in crisis. Swift was probably hoping that some readers would recognize themselves in the essay.