Not much physical danger unless the guy or gal you're working for is physically abusive. And hey—that's not that far outside the realm of possibility, considering some of these actors' tempers. We hear that Amy Adams packs a mean wallop.
While there is nothing specifically in your job description to worry about, you will be moving around almost constantly, so you'll be more at risk of something bad happening than, say, someone sitting in one place for eight hours a day. Getting on and off of planes, driving your celeb all over town, having to maneuver your way through throngs of pushing, shoving fans…these are all situations that could conceivably result in injury. Not to mention your celeb himself, who is walking all over you.