Stress
Typically speaking, epidemiologists are scientists, and tend to view their work as scientists do. Sure, the little red dots on the heat map are actually people, but you're working with data, and data is a known, unchanging quantity. For most days, you'll do what you've set out to do, go home, enjoy your above-average salary, and chill out on the couch watching junk TV for three hours before bed.
Of course, dealing with outbreaks can also mean travel...and not the bed and breakfast, "why yes, I'd love another croissant to go with my espresso" sort of travel, either. If you're on the road, the stakes have likely raised. You're not going to have it as bad as the doctors, but you're going to be smart enough to know that your work matters and that a clock is ticking.
Additionally, you'll be handling dangerous materials a lot of the time. Samples from infected humans or animals, tainted drinking water, decaying tissue. The range is wide and gross, so you'll have to be very well-versed in safety techniques. Occasionally, you'll even come across something outside of your experience, which will require some in-the-moment snap decisions. You got this, though.
If that's the sort of thing that sounds like it might stress you out in an unhealthy way, don't worry. There are plenty of epidemiologist gigs that never make you leave the office, and yes, they're a big help, too.