FBI Special Agents aren't always laying siege to fundamentalist Christian compounds or negotiating with hostage-taking bank robbers—there's a fair amount of paperwork involved, too—but they're the first into the fire when those types of situations arise.
If you're committing any type of federal crime, you can bet that the Feds are going to be the ones coming for you. As such, being an FBI Special Agent is one of the most dangerous jobs around. At any given moment, a Special Agent could be called in to handle a hostage crisis, or arrest a known serial killer or domestic terrorist, or investigate a bomb site, none of which are exactly desk jobs.
Not convinced? How about this: Special Agent Joseph D. Pistone, the inspiration for the film Donnie Brasco, spent six years infiltrating an organized crime family. He had to pretend that he was a completely different person, and if anyone ever suspected who he really was, he would've been killed instantly. So yeah, there's some risk of physical danger. But "Danger" is your middle name, right?