You may be able to remember your name in the desert, but the monster can't—because he doesn't have a name. Without a name—a label—there's really no way to make sense of him. Is he a hero? A villain? Does it matter what people call him, or what he calls himself? Even when he learns language, he doesn't get a chance to open his mouth before people run away screaming. So how do you communicate when no one can stand to look at you?
Acquiring language not only gives the monster a sense of his own humanity, but it forces him to come to terms with his alienation from society as well. Like the monster, language can be good and bad.
The monster's namelessness is the reason that he is alienated, more than his ugliness.