And they are gone: ay, ages long ago These lovers fled away into the storm. That night the Baron dreamt of many a woe, And all his warrior-guests, with shade and form Of witch, and demon, and large coffin-worm, Were long be-nightmar'd. Angela the old Died palsy-twitch'd, with meager face deform; The Beadsman, after thousand aves told, For aye unsought for slept among his ashes cold.
And then, after all this build-up: Poof! They're gone.
There's a big narrative shift in this last stanza. This entire time the story has been unfolding in front of you, so much so that the speaker has directly addressed the characters and reader at different points to direct them to different things as they're happening. Here, though, it's like we've skipped forward a thousand years, and find out that everything we've been seeing actually happened way in the past.
Funnily enough, you don't end the poem with Madeline and Porphyro, even though the poem is presumably about them. All you hear about them is that they "fled away into the storm," which is actually kind of ominous because you never learn whether or not they live happily ever after or even if they get to that house of Porphyro's across the moors. For all we know they just get sucked into the "elfin-storm" for forever.
Instead, we hear about the other people back in the castle: the Baron (who's never even been mentioned before) and all of the party guests were "be-nightmared" and spent all night dreaming about "witch, and demon, and large coffin-worm." It's strange that, during the time when Porphyro and Madeline were off doing their "young people in love go make a life together" thing, everyone around them was dreaming about death. This makes the couple's end feel even more uncertain.
Some people had worse problems than nightmares: both Angela and the Beadsman died that night.
This isn't totally unexpected—Keats had gone out of his way to make us feel like these people were going to kick the bucket any moment—but it's still pretty jarring that what should be a love poem ends in death and ashes.