Physical Danger
Rock bands are known for participating in more…socially-unacceptable behavior than most people. Such unacceptable behaviors are known to include all sorts of substances and maybe even some danger-seeking tendencies. Handling the finances for that type of crowd might feel a bit like throwing yourself in harm's way.
Especially if you're working for someone like Ozzy Osbourne, who seems like the type to bite your head off. (But you're not a bat, so you have less to worry about.)
But there are a lot of different, and presumably safer, bands you can work with. Maybe get yourself aligned with a mellowed-out jazz outfit, or an all-flute ensemble, or with old rockers who are past their hard-partying days. Whatever you decide, just make sure you don't bite off more than you can chew (which is advice Ozzy should also heed).