At last, the speaker gets around to telling us what actually happened when he heard the learn'd astronomer, and so on and so forth.
And what happens is…he comes down with the flu. Well, not really. But he does feel "tired and sick." He doesn't make it very far into the lecture before this happens. Patience is not his strong suit.
It might be an exaggeration for him to call himself "sick," as if his frustration with the lecture has led directly to physical illness.
The word "unaccountable" is a clever response to the blizzard of numbers and charts presented by the astronomer. "Unaccountable" means "for no apparent reason." He cannot explain why he feels this way. His inability to explain contrasts with the astronomer, who believes he can explain even such mysteries as the night sky. Plus, "unaccountable" obviously contains the word "count," just in case we had forgotten that the speaker really isn't a fan of the math.
Till rising and gliding out, I wander'd off by myself,
And there he goes. The speaker gets up and leaves the lecture hall, probably annoying everyone in his row in the process. Dirty looks abound.
But, to hear him tell it, the speaker's exit is as graceful as a ballet dancer. He literally "glides" out of the room. Classy.
Our speaker is obviously the kind of person who would leave a movie after ten minutes if it looked like the plot wasn't going anywhere. He doesn't like to waste time.
He wanders around by himself: where's he going?
In the mystical moist night-air, and from time to time, Look'd up in perfect silence at the stars.
Outside the lecture room, all is dark and quiet. The air is "moist" with dew or humidity, which sounds like a warm, sticky summer night to us. Every so often he looks up at the stars but, unlike the astronomer, he doesn't try to explain them or even express their beauty (though, in a sense, that's what the poem does).
"Mystical" means unified with God or nature. "Mysticism" is a solitary religious activity – the very opposite of the communal, educational atmosphere of the lecture hall. To be a mystic, you don't need to have an education. You only need to have your spiritual sense open to the world.
If you think "mystical moist" gives a slightly sexual or erotic impression, we're right there with you. Whitman frequently uses expressions that can be interpreted in either a sexual or non-sexual way.
The speaker's silence obviously stands in contrast to the astronomer's jabbering on and on about equations and numbers. The speaker enjoys the stars rather than trying to explain them. For a poem with "astronomer" in the title, we're surprised that the word "stars" is not used until the very end.
Whitman's choice of words in this poem is brilliant. Because we can never fully understand the stars, all knowledge of them must be incomplete or "imperfect." Silence, on the other hand, seems to capture their beauty "perfectly."