Physical Danger
The worst part about being a safety inspector is you're the one people expect to say if something is safe or not. That means you're the one who has to test it.
Picture thinking to yourself, "It would be really bad if this wall weren't properly bolstered," then testing that wall to find out it's indeed a little shaky on its supports. Make sure you always know where the nearest exit is, just in case you need to run there as fast as you possibly can.
Your workspace will be a lightless tunnel of dirt and rock—it isn't the safest place in the world to begin with. These caverns may also be filled with gases that could cause an explosion if you're not careful. Or, drilling machines that aren't working properly could suddenly decide to work when you're standing in front of them. Taken together, the whole thing seems like a booby-trapped set from an Indiana Jones movie.
And if things are really out of shape, the whole thing could collapse on you like that couch fort you made when you were seven—only instead of pillows it's made of thousands of pounds of stone.
This is why your job is so important; the regular Joe and Jane Miner are working in these conditions on a daily basis. That's also why you need to maintain your own safety while you do it, because with one wrong move you could suddenly become a fossil.