Physical Danger
The majority of your clients won't be heavyweight boxers (or at least, not heavyweight boxers any more…hopefully), so you don't have to worry about getting into a many physical altercations with your patients. You may encounter patients who are resistant to treatment and, frankly, who don't want much to do with you.
Depending on the severity of the situation, they could get a little hostile. In that case, you might just have to get physically involved with a patient (for example, maybe holding an arm down for an injection).
The most dangerous aspect of being a doctor is stress, which can have physical repercussions. It's important to make sure that, as a doctor, you worry about your own mind, body, and spirit in addition to all of your clients. And make sure you wear some good shoes for all that standing and walking you'll be doing.