Physical Danger
Since this is a law enforcement position as much as it's an accounting position, the dangers you will face will be similar to those faced by other civil servants who carry a badge. The IRS special agent is probably the most dangerous job an accountant can have. You'll occasionally deal with people who are armed, and you'll be licensed to be armed yourself. It's a good bet the junior year math teacher who pushed you into a career in finances never considered that.
To be honest, no IRS special agent has ever been murdered in the line of duty, although four have died while on the job. That doesn't mean you won't face threats from nefarious corners; after all, the people you will be dealing with have broken the law (or many laws, depending).
Much of your job will be spent inside a government office building, but you'll get out on the street from time to time to interact with witnesses and interrogate suspects. You may even end up under cover. You could appear to be a mild-mannered tax accountant when really you're a mild-mannered tax accountant with sweet, James-Bond-type moves. The important thing is to stay aware of your surroundings.