The speaker of "Richard Cory" is a collective we. The poem describes them as "we people on the pavement." These people live "down town," and they all admire one Mr. Richard Cory, who visits from time to time.
There are two ways that we can interpret this "we." Either the "we" is a sort of Greek chorus, a voice of many telling the tale of Richard Cory. Or, the "we" is the voice of an individual, speaking not only for himself, but for his fellow townspeople as well.
Whichever interpretation you choose, one thing's for sure: there's an immense distance between the people on the pavement and Mr. Cory. He's a solitary dude among a populous "we." More than that, he's separated from the speaker by his wealth, his fashion, his manners, and (it seems) his good looks. He's a model in all of those things, someone who is the object of the speaker's admiration. While the speaker is going without meat, R.C. is setting hearts a-flutter just by saying hi.
That makes his death at the end all the more jarring. Should the speaker really have envied Richard Cory all along? As readers, we're left with the same realization as the speaker at the end of the poem: appearances can be very deceiving.