Physical Danger
The dangers faced by each workplace safety inspector are directly related to the type of workplace being inspected, which is probably pretty obvious (source).
A lot of it has to do with amount of equipment, or lack thereof. If you're examining worker wage records or leading an employee seminar on sexual harassment, the worst dangers you'll face are awkward questions. Interviews with management should be fairly safe as well, unless your critique of their business practices leads to chairs and other objects being tossed at your head.
Once you start throwing mining drills, gas ovens, and power plants into the mix (hopefully not at the same time), the danger quotient goes way up. The best way to deal with the various hazards is to know your way around them, which means inspecting in the industry that you know best.
You'll also want to use the proper safety equipment for the job being inspected. YOLO may be a useful attitude for a professional athlete, but it's not really what people are looking for from the person charged with making sure everyone's safe.