"Avow" is an important word here. The speaker wishes to make a strong declaration of fact, which suggests that he wants to be certain, rather than skeptical, about something.
You are not wrong, who deem That my days have been a dream (4-5)
The speaker doesn't say that the woman is right, only that she is not wrong. The speaker seems unwilling to accept the skeptical position that life is all a big dream. There's a wee bit of hesitation there.
All that we see or seem Is but a dream within a dream (10-11)
The speaker sure seems pretty convinced that life is just a dream within a dream. But skepticism is supposed to be about uncertainty, so this doesn't seem so skeptical to us. Can you really be certain about uncertainty?
I stand amid the roar Of a surf-tormented shore, And I hold within my hand Grains of the golden sand— (12-15)
The speaker definitely emphasizes what he's doing. Note the way "I stand" and "I hold" are at the very beginning of their respective lines. It's almost like he's trying to convince himself that these actions—standing at the beach, holding sand—are actually happening, and not just part of a dream within a dream.
Is all that we see or seem But a dream within a dream? (23-24)
The speaker's final words are a question, not a declaration. He's being totally skeptical here, refusing to give an answer to the question of whether or not really is just a dream within a dream.