This poem actually has a bunch of speakers. Maybe the best way to sort them out is to divide the poem into its two major parts. Ready? Okay, then…
Speaker 1
The first part is a kind of introduction—five stanzas that set the scene, and tell us where we are and who we're dealing with. The speaker for this first part is an anonymous, third-person narrator type. We don't learn much about who he is, or what his perspective on this whole crazy business is. At the same time, because of what comes after, his lack of personality really stands out. In a way, his calm, objective look at things makes the Lotos-eating sailors seem even loopier.
Speakers 2-???
After the first five stanzas, the sailors take over. They are the speakers for the rest of the poem, as they sing their "Choric Song." We don't hear much about how many of them there are, or how old they are, or what they look like, but they do tell us a bunch about their feelings and desires. We hear all about how tired they are, about how they don't want to go back to the hard work, the rowing and the struggling of their life on the sea. We even get to hear a little about their past. They fought in the war in Troy, they've been travelling ever since, and they have wives, children, even slaves waiting for them back in Ithaca, their home island.
We think having all these speakers makes "The Lotos-Eaters" feel more like a play than a poem. In fact, the fact that Tennyson talks about a "Choric Song" is probably an allusion to ancient Greek theater, where a group of actors (the chorus) would have explained the moral of the play.
This poem, at the end of the day, is a study in contrasts, and the speaker situation is no exception. We have both a calm, collected, and mildly-detached narrative speaker, as well as a more forthcoming (if totally blissed out) collective speaker. Each represents a different perspective of reality, and it's up to the reader to determine who's sane and who's gone totally round the bend.