(Click the character infographic to download.)
At the beginning of the book, wizards all over the land are celebrating because Voldemort has been defeated. So, we learn from the start that he is one unpopular wizard. Voldemort's reign of terror that has oppressed so many wizards for eleven years has finally ended at the start of the book. Even though he's known for being incredibly powerful, a mere baby defeats him when he tries to kill it. As Hagrid explains to Harry when he comes to deliver his Hogwarts admission letter, Voldemort was a "wizard who went… bad. As bad as you could go. Worse. Worse than worse" (4.99).
Voldemort inspires such fear in his opponents that they refuse to call him by name, as though using his name will conjure him up or alert him to their presence. The only people who aren't afraid to speak Voldemort's name are Dumbledore and Harry. Dumbledore says he "ha[s] never seen any reason to be frightened of saying Voldemort's name" (1.41) and Harry says he's not saying it out of courage: "I'm not trying to be brave or anything, saying the name […] I just never knew you shouldn't" (6.159).
While Voldemort is present as an ominous shadowy kind of being, he doesn't show up "in person" until near the end of the book. Even then, he's using the body of one of his followers, Quirrell, to live a miserable, parasitic, hidden existence. We find out that he's been killing unicorns and drinking their blood to sustain himself. Lovely. Firenze explains:
"The blood of a unicorn will keep you alive, even if you are an inch from death, but at a terrible price […] you will have but a half-life, a cursed life, from the moment the blood touches your lips." (15.155)
Voldemort is willing to pay this "terrible price" and take on this "half-life" in order to mount his comeback and return to power. Of course, he thinks he's going to get hold of the Sorcerer's Stone and be able to make the Elixir of Life. If anything can remove the really bad side effects of unicorn blood, this immortality drink should do the trick.
When Voldemort's doesn't get the Stone, he's temporarily defeated. Quirrell is killed and Voldemort must retreat, to seek out another body to inhabit. Wiser characters know that Voldemort hasn't been defeated for good. Dumbledore tells Harry:
"[W]hile you may only have delayed his return to power, it will merely take someone else who is prepared to fight what seems a losing battle next time – and if he is delayed again, and again, why, he may never return to power." (17.112)
Series readers will know, however, that he will just resort to even more sketchy means to return next time.