In describing fortune as "delusive" here, the speaker suggests that our hopes and dreams mislead us. We may make it big, we may grow rich, but then: boom! We tumble down.

Not only do our hopes end, but so does love. All of those politicians, or statesmen, who had hordes of people following them suddenly find that no one cares about them, once they're not important or powerful. There isn't much to look forward to in life, in other words.

The speaker is talking about Cardinal Wolsey's end here. All of his hopes and dreams have come to nothing. He's lost favor with the king (and with everyone else), he's made mistakes that now sting him, and he's sick. The speaker uses Wolsey's end as evidence for why we shouldn't count on things working out for us.

Even those bookworms—scholars—don't have such an easy time. They pursue the scholarly life because of the promise of satisfaction and reward. Instead, they find bad things—envy, poverty, and even prison (if they cheesed off the king back then with their ideas, we were likely to end up in jail).

The speaker is referring to the Swedish King Charles XII here. He was a big man, with huge ambitions and dreams. Those ambitions and dreams were very sadly diminished by the end of his life. He's exiled to a small fortress on a barren shore. Not only that, but the only thing he's remembered for ("his name") is that he ended up badly.