Stress
Weekends aren't necessarily a guarantee. That alone makes this job stressful. Customs and immigration inspectors need to anticipate pulling in overtime, which includes holidays, in addition to the occasional 24-hour on-call shift. It can be pretty stressful to live a life where everything you plan might be ruined by an emergency at work.
Going through customs is stressful enough for travelers, so imagine how is it is for those regulating it. Speaking directly to those coming into the country—people who are often already tired from a long day (or possibly days) of travel, or who are frustrated because they speak minimal English—can be hard for the inspector and the traveler.
All in all, it's important to not only provide the support that travelers need and communicate with them clearly, but to ensure that the process is going smoothly. Yes, you need to be constantly aware of any and all red flags that may be a threat to national security. Yes, you need to be a human polygraph and have X-ray vision and telepathy. But you also need to have insane amounts of patience so that you can speak to exhausted, harried, smelly travelers without biting their heads off.
If the thought of having to do all that stresses you out...it's not too late to become a beekeeper, you know.