In "To an Athlete Dying Young," we get a pretty clear sense of our speaker and you might even find yourself feeling a kind of connection with him. This is happening for a couple of reasons.
First, it's pretty clear that the speaker has some firsthand knowledge of what this athlete was like. He was there when the athlete won his big race: "We chaired you through the market-place" (our empahsis). The speaker was also there at the funeral: "Shoulder-high we bring you home" (again, our empahsis). It seems likely that the athlete and the speaker were even friends, since those chosen to carry the casket usually have some close connection to the deceased. So, by the end of line 8, we have a pretty good sense that this speaker has personal knowledge about the tale he is telling. It makes us trust him (and we assume it's a him, with no evidence to the contrary). It makes us more willing to listen.
Secondly, it's important to consider to whom the speaker is talking. The title ("To an Athlete Dying Young") tells us that the speaker is talking to the dead athlete: "The time you won your town the race." Fair enough—we are all familiar with movie scenes of people in graveyards talking to loved ones who have passed away. But the speaker doesn't just talk to the dead athlete. He also talks to the mourners. He tells the mourners to "set" the casket at the tomb's doorway and hold up the athlete's trophy ("The still-defended challenge-cup").
There is one other group that the speaker addresses in this poem that really makes us feel like part of it: the speaker talks to us, the readers. When the speaker refers to "you," we know he is, in the context of the narrative, referring to the athlete. But when we see and hear the word "you" over and over, it's hard not to get the sense that someone is speaking to us. Suddenly we are forced to consider our own mortality and the uncertainty and fragility of our own lives—a valuable, but incredibly terrifying, exercise. Thanks for nothin' Housman.