Gardeners have it easy in this area, at least. There certainly is some degree of stress, but not nearly as much as in other lines of work. Most stress does not come from the work directly, but from rushing about town to get to various jobs in a hurry. Most gardeners do not have the luxury of going to a single location to do their work. They need to go to their yard to get their pickup or truck, and then travel to the jobs they have on their slate. The time spent in travel cuts into time that can be billed while on the job. That’s why most speeding ticket violators in this country are gardeners.*
*Totally made-up statistic.
The stress of running a gardening business is like that of running any other business. Bad weather can be a bit stressful, or fun, depending on how you look at it. (Are you a "my shoes are half full of rainwater" or "my shoes are half empty of rainwater" sort of person?) Altercations with angry clients should be (but are not always) relative to the quality of work. Those who do a good job should not have too many unhappy clients to deal with.