Like the main character, the music in Spellbound doesn't seem to know what it is. Sometimes it's big, sweeping, orchestral, and romantic, decked with flowers and opening every door in Constance's head. Then at other times it's eerie and creepy, moaning as Ballantyne freaks out about lines on white for the twentieth time. It's a split-personality score for a movie about a split personality.
Composer Miklós Rózsa says that this was just the effect Hitchcock and producer David O Selznik wanted. They called for "a big love theme coupled with the strange sound for the paranormal" (source).
Rózsa, a Hungarian immigrant with a bunch of film scores under his belt, including the famous The Thief of Bagdad (1940), composed a big, orchestral love theme, and then added additional music using a new electronic instrument called the theremin, which would go on to be one of the standard sounds of Hollywood horror movies. (Here's a sample of Leon Theremin playing his own instrument, so you know what it sounds like.)
Rózsa won an Academy Award for the score… but as you'd expect from a split-personality score, he wasn't the only one responsible. In fact, Hitchcock, Selznik, Rósza, and music editor Audray Granville all tinkered with and rejiggered the score.
The end result is a jumbled-together product of editing, re-editing, re-re-editing, and re-re-re-editing. You can't tell who did what—just like Ballantyne can't tell that he's Ballantyne (source).
Even though he was partially responsible for it, and even though it won an Academy Award, Hitchcock wasn't happy about the score, which he felt was so loud and boisterous that people would get distracted from the action.
There is something to that; there are often times in the film where the score seems to be leaping up and down, shouting, "Look at me!" In any case, Hitchcock found it annoying. Partly as a result, he and Rózsa never worked together again