If you're working with potentially dangerous tools and elements, yes, there is going to be a certain amount of stress involved. And if, within the realm of that science, there is some public backlash, there will be a different sort of stress that you'll have to learn to deal with.
That said, taking all the necessary precautions to ensure your safety and the safety of your coworkers ensures that your possible anxiety about that part of it will be kept to a minimum. And about the other aspect of the job? Your family's and friends' concerns that you glow in the dark? You’re going to have ease their trepidation with your incredible knowledge and understanding of nuclear science.
You're not just a scientist when you go into this field; you're also a public relations mouthpiece. You know what goes on in the lab, the plant, the hospital, or the classroom and it's up to you to communicate that to the folks around you. Nuclear science is here to stay but if you can't convince the people you're trying to persuade that it's actually a pretty safe industry, well, at least you know it, and that in itself will relieve your own apprehension.