Physical Danger
As an inspector, you may be dealing with people who are in violation of customs or immigration laws with varying degrees of severity. This means you may have to whip out those handcuffs and detain people, which in turn may involve preparing them for legal action or for immediate deportation. Handling someone who may be facing legal action or deportation can be a sticky (and physical) situation that requires some intense training. So don't scrimp on the push-ups.
In addition to people, you may also be responsible for seizing any illegal contraband arriving or leaving the U.S. You're probably not going to be able to take a leisurely stroll onto someones ship and take their property while they nod politely at you. This is why the feds stress that their inspectors be in great shape. It's important to be able to go into a situation and protect yourself.