In The Man in the High Castle, "the man in the high castle" is Hawthorne Abendsen, the author who lives in a fortified house that is called—wait for it—"The High Castle" (6.116). However, Abendsen no longer lives in that "High Castle," ever since he developed of phobia about his elevator (15.69). So the Man in the High Castle doesn't actually live in the High Castle—which seems to fit in a story with an author who didn't write his book and all those fake "antiques."
But Hawthorne Abendsen isn't the only person living in a High Castle. There are two Nazis who have some relation to castles: Hermann Goring and Reinhard Heydrich. When the Japanese consulate gives a lesson about the potential leaders of the Nazis, they note that Hermann Goring is said to live in a castle with all his treasure (6.161). As leader of the SS, Heydrich controls something called the Castle System (12.60), which is probably related to certain real-life places where the Nazis taught or planned to teach elite Nazis. Now, Goring and Heydrich seem like pretty bad Nazis; but at the end of the book, we learn that the world might be slightly better if Heydrich, the worst Nazi, takes over.
And so we have two radically different groups of people, both of whom have relations to "castles": Abendsen, who writes anti-Nazi fiction and the terrible Nazis. Though these groups seem very different, the book seems to show that the Men in those High Castles might lead the way to a new world, for better or worse. So when Dick uses the vague phrase, "The Man," he's probably doing that on purpose.