In the first lines of the poem, you have no idea that love, or specifically "desire," will be compared to hot, scorching flames. You might be thinking "fire" means a comet hitting the earth.
From what I've tasted of desire (line 3)
For someone who seems so worldly-wise, we're surprised when the speaker claims he has only "tasted" desire. We think of desire as something that you devour in one giant bite, or, rather, as something that devours you. He is being modest here. He claims that even his limited exposure to this huge force of love is enough to get a sense of its power.
I hold with those who favor fire. (line 4)
The speaker shifts back to his "judge mode." He can analyze his own experiences with detachment.