Remember, rudiments are present only in a developing organism—they're evolutionary leftovers that appear briefly and then disappear. Vestiges, on the other hand, are evolutionary leftovers that stick around even in an adult organism.
"Ontogeny recapitulates phylogeny," is a phrase that get's thrown around in the biology community. Ontogeny is a fancy word for development, and phylogeny refers to an organism's evolutionary history. This phrase is saying that an organism's development mimics its evolution—that is, the stages it goes through from embryo to adult resemble various ancestors that eventually evolved into the modern organism. Taken literally, this would mean that a human embryo starts out like a little fish, and moves on to an amphibious stage, and then a reptilian-like stage before becoming distinctly human.
This idea was popularized by a German zoologist named Ernst Haeckel in the later part of the 19th century, and has since been fully discredited. Nonetheless, the idea was widely influential and believed for so long that it still pops up now and then. Don't be fooled. Although we most definitely share an ancestor with fish, amphibians, and reptiles, human embryos most definitely do NOT go through fish, amphibian, and reptile stages of development! Development is a credible source of evidence for evolution, but not because it literally mimics an organism's evolutionary history.
A common misconception about common ancestry between humans and other primates is that humans descended from monkeys, or that monkeys are our ancestors. Both humans and monkeys (and gibbons, gorillas, and chimpanzees) descended from a common ancestor, but that ancestor was not a monkey, gibbon, gorilla, or chimpanzee. Or a human, for that matter! The ancestor likely possessed traits that all these primates share, but it also likely possessed its own unique traits. Saying humans are descended from monkeys is like saying that you are descended from your cousin—it doesn't make sense. What does make sense is that you and your cousin share ancestors (your grandparents); likewise, humans and monkeys share an ancestor.
Don't confuse homologous structures with analogous structures. Homologous structures, by definition, are inherited from a common ancestor. They may or may not be used for the same functions in all the descendants that possess them. Think about the phalanges (finger bones) in a human and a bat. Phalanges are homologous in humans and bats because they both inherited phalanges from a common ancestor. Humans use their phalanges to hold and grab, to pick their noses, or to tickle other people. Bat phalanges have evolved to make up part of their wings; they essentially use them to fly.
Analogous structures may have the same function, but have different origins. A bat's wings and a bird's wings are both used for flight, but they evolved independently—wings were NOT inherited from a common ancestor, so wings in bats and birds are not homologous.