If you really wanted to get carried away with symbolism, you might compare those columns to prison bars. At any rate, they represent the kind of artificial order that society opposes on the natural world.
where he lectured with much applause in the lecture-room, (line 4)
We don't learn that the speaker is in a crowded lecture until this point in the poem. He feels alone and isolated despite the crowd. The other people are mentioned indirectly, through their "applause," but otherwise we know nothing about them.
How soon, unaccountable, I became tired and sick; (line 5)
The speaker is a sensitive soul in the crowd. He's the only one who reacts so negatively to the lecture that he must escape. That is, we assume that the poor astronomer doesn't have everyone running for the exits.
Till rising and gliding out, I wander'd off by myself, In the mystical moist night-air, (lines 6-7)
Despite his solitude, he feels much less alone once he goes outside the lecture room. The line separating humanity and the natural world dissolves, and he feels a close connection with the night, symbolized by the vapor (moisture) that permeates the air.
and from time to time, Look'd up in perfect silence at the stars. (lines 7-8)
He does not need to speak for the sake of speaking. In some mystical traditions, silence is a "perfect" language, or a language beyond words. It's true, though: the stars make for great company.