There have been lovers who thought love should be So much compounded of high courtesy (24-25)
Love made of "high courtesy" is love filled with quotes and chivalry, says the speaker. It makes love a job, one that requires study and manners. Since the speaker believes that anything beautiful requires labor, love can be no exception.
We sat grown quiet at the name of love (29)
Love is so powerful that even the word makes them suddenly go quiet. This marks a turning point in the poem, as the tone changes from conversational to sorrowful, and the imagery takes a turn towards the pensive—all because of love.
I had a thought for no one's but your ears: That you were beautiful (35-36)
The speaker is addressing one of the women, and it's the first time we see hint of his feelings towards her. It's a hint of the heartache we see in the final lines of the poem.
[…] I strove To love you in the old high way of love (36-37)
He tried to love her in a chivalrous way. This is the type of love the poem lauds, the type of love that involves manners and book-learning. Apparently, love is one of those things that, even with effort, doesn't always work out.