Science Fiction?
You could probably write a pretty good short paper on this question: Is The Man in the High Castle science fiction? In fact, this issue becomes a question inside this book, when Paul and Betty Kasouras argue over whether alternative histories count as science fiction (7.51-3).
But we're going to say that The Man in the High Castle is science fiction for the following reasons:
First, note that we're not calling it an adventure, even though Baynes's story is about spying, and we're not calling it a drama, even though Childan's dilemma is classic drama. If we're talking about genres of plot (drama, quest, adventure, mystery, etc.), we can say that The Man in the High Castle has a few different plots. But no plot dominates. Every time we think something like "Childan's drama is the most important," we reread the book and discover that someone else's story is even more important. (Because they're all important.)
Second, the book starts with a classic "what if?," the same sort of question that starts off lots of other science fiction works. "What if you could make life?" asks Frankenstein. "What if we could see the effects of time?" asks The Time Machine. "What if Arnold Schwarzenegger was an awesome time-traveling robot?" asks Terminator. Here, the question is, "What if the Axis powers won the war?"
(Note: it could also be a history book or some other form of nonfiction with that question as its starting point. But, hopefully, we all see that this is fiction.)
Third, this book includes two weird, non-realist elements: Tagomi "slipping" into a different timeline and the I Ching telling everyone about the other realities out there. We may discover that this is actually the way the universe works, but for now, those elements make the book seem science fictional to us.
Now, we're just talking about this one book; whether alternate history is part of science fiction is still an open question. If you don't believe us, check out the discussion here—and join in.