I said, 'A line will take us hours maybe; Yet if it does not seem a moment's thought, Our stitching and unstitching has been naught. (4-6)
To "stich and unstitch" a line is to re-work it over and over until it's perfect. Yikes—Yeats says this can take hours, but he also says it should look like it took only a moment or it isn't a good line. That's a tall order.
For to articulate sweet sounds together Is to work harder than all these, and yet Be thought an idler by the noisy set (10-12)
The "noisy set" are people that let everyone know what they think: that writing poetry is idle, far more idle than manual labor. But the speaker says that articulating "sweet sounds" is managing to put music into words, and he thinks that kind of work is harder than scrubbing floors.
There have been lovers who thought love should be So much compounded of high courtesy That they would sigh and quote with learned looks Precedents out of beautiful old books; (24-27)
Books used to hold the key to love, too. The speaker clearly has high respect for literature and the art of writing, and considers any wooing done with "high courtesy" to be done by those who are well-read.