Stress
Let's check the big one off the top: train whistles are migraine-inducing devices of audible torture, and they're just one of several dozen loud noises you'll have to deal with in a given day. You heavy metal or hardcore electronica fans will probably do just fine with this.
Another great stressor in the life of a rail inspector is the copious amount of travel you'll embark upon. Even the main government website has a big warning on their main job overview page: "Railroad Safety Inspectors travel frequently which requires being away from the official duty station for several consecutive nights." It says "official duty station," but it might as well say "family and friends," because that's who you're going to miss the most.
Recognize that the long hours and distances will weigh on you from time to time, so make sure to log in those vacation hours whenever the opportunity arises. On the bright side, with all those days on the road, you'll finally have something in common with your favorite band.
There's also the stress that comes with knowing one mistake could mean a train goes off the tracks, or two trains smash into each other, or a car gets hit due to faulty signals, or hazardous materials are spilled into a major river. But since this is a safety job, we assume you're already aware of that. It's unlikely that those stressors will ever stop weighing on you, so be ready to deal with those doubts and bring your best work ethic with you to the station every day.