Men? There are no human beings in the poem, so what's this about "men"? Well, the first word of the poem is "he," so we know that this eagle is a guy. But, really, how does the speaker know this? Clearly the speaker has chosen to assign a gender to the bird, and he must have had a reason for doing so. The eagle represents a kind of masculine ideal of power, solitude, and gracefulness. Just as a superhero seems most powerful when he holds his powers in reserve, the eagle awes us simply by standing there. He lives apart from society but can descend to our level when he needs to. You could easily read the eagle as an old-fashioned portrayal of a great and noble man. Nowadays, most of us wouldn't bat an eye if the eagle were called a "she."
The eagle is a fantasy of male power held in reserve. The eagle is located at the center of the sky, and he assumes a dominant position relative to the old-looking, "wrinkled" sea, which "crawls" before him as if in supplication.
The poem has nothing to do with gender. The speaker gives the eagle a male gender simply because it would have been customary to do so.