From the very first page, music is constantly in the background. It’s not Stephen’s primary artistic passion, so it never really steps to the foreground, but it’s always a lingering presence. Stephen is a singer; we don’t know how talented he is (he is asked to perform several times, which indicates that he must be pretty good), but it’s never a central part of his identity, as far as we’re concerned. However, his "sensitive nature" is very receptive to musical cues, and he often thinks of language in terms of its musicality and rhythmic nature. He refers to phrases making up "chords" with words, an idea that combines the concept of musical harmony with poetic beauty.
Music appears at several key points. For example, when he is about to leave the Director’s office in Chapter Four (on the brink of deciding whether or not to join the Jesuits), the priest’s "mirthless" response to a sudden burst of music from the street shocks Stephen, making him realize that he could never become a priest himself. Later in the chapter, Stephen imagines an "elfin prelude" that expresses his excitement at the prospect of going to university. For Stephen (as for many people), music is tied to a level of non-verbal, almost primal experience of emotion. It relates to his more intellectual poetic activities, but also to his spontaneity and his immediate reaction to the outside world. Joyce himself was really interested in trying to represent music in words; we see him do this both in Finnegan’s Wake and in a truly monumental episode in Ulysses (if you’re interested, it’s Episode 11, "Sirens").