First and foremost, the rhyming is the most obvious sound effect in the poem. Each stanza is carefully knit together with alternating end rhymes (the lines' rhyming last words). A little A action, then some B, and so on and so forth.
There's nothing more satisfying than having a puzzle fit together, and that's exactly what each of these stanzas is. The rhyme scheme sets up a musical expectation for us. Once our ear picks up the pattern, it expects to hear it again. In this case, it totally does. There isn't a hair out of place.
There are a few subtle sound tricks Yeats uses, too, that make for a smooth-sounding poem. He repeats the phrase, "I will arise and go now" at the beginning and end of the poem. Repetition does a couple things here.
First, like rhyming, it allows our ear to make a satisfying sound connection. Second, the declarative nature of the phrase (the speaker will do something), gives the poem an important tone. It almost rings as "I shall!" This guy is determined. We're ready for him to take charge.
Another way to sound confident and important in poetry is to use inversion (switching the order of words in a sentence). So when Yeats writes, "And a small cabin build there" rather than "and build a small cabin there," our ears perk up. Face it, folks. In this compact little poem, Yeats uses all sorts of devices to get us to listen up.