Stress
Let's face it, this is a great job that would impress just about anyone at a cocktail party. So what's there to be stressed about? Nothing, except the expectation to come up with mind-shattering discoveries about the universe. No biggie.
The most likely cause of stress for an astrophysicist is the pressure to earn funding for their research projects. In a field where many are competing for grant money, not every project can be funded. If you're unlucky enough to be repeatedly left in the lurch, that's going to take a toll on the amount of time you have to get real work done.
Astrophysicists also tend to work full-time, but if you're wrapped up in a search for answers to celestial questions, there's always the risk that you'll have trouble leaving your work at the office or laboratory. You can be sure that winners of the Nobel Prize pored over their research more than just between the hours of 9:00AM and 5:00PM.
That said, in between their intense periods of research and exploration, most astrophysicists receive benefits that include a decent chunk of vacation time. If you're working long hours and agonizing over findings that just don't seem to add up and you think you just can't take much more before you snap...just take a holiday.