Only two things are certain in life: death and taxes. The first means the end of your career. The second means that, barring some apocalyptic end to government and its need for funding (and politicians like money) there will always be a place for a good IRS agent.
Staying in the business is therefore mainly up to the individual and how well they can handle the pluses and minuses (and multipliers and dividers and “carry the ones”) of the business. Careful, accurate, level-headed auditors should be assured of a stable career.