"Like A Rolling Stone" owes a great deal to the traditions of folk, country, and blues that influenced Dylan before he went electric, and the name is a tribute.
Though Bob Dylan followed folk history and folk music closely, he was a very critical listener and there was little that he loved. His favorite songwriter was Hank Williams, whose 1949 release "Lost Highway" (actually written by Leon Payne) tells a story of a delinquent boy with nothing left, and starts with the lines "Like a rollin stone/all alone and lost." This is one of Williams' signature songs, and the 1965 documentary Dont Look Back features a great scene of Dylan enthusiastically singing the song while Joan Baez helps him remember the lyrics. Dylan once said "I started writing songs after I heard Hank Williams."
Muddy Waters' 1950 "Rollin' Stone" is one of the most famous songs to connect the old acoustic blues of the Mississippi Delta to the electric blues, the newfangled 1950s Chicago genre out of which rock 'n' roll first emerged. Waters was a huge figure in the electric blues revolution, and everyone from Chuck Berry to the Beatles was inspired by him. But no one was more inspired than those young Brits who actually named themselves after Waters' song: The Rolling Stones.
When Dylan released "Like A Rolling Stone," the so-called British invasion (the extreme popularity of British bands like the Beatles and the Stones in the U.S.A.) was at a peak. The Rolling Stones were some of the biggest rising stars, and people couldn't help but wonder when they first heard about the song: is Bob Dylan trying to one-up the Brits? Is he mocking them? When the public actually heard the song, however, nothing was so obvious. Bob Dylan seemed to be entering the world of the Rolling Stones with both reverence and skill. Eventually, the Stones and Dylan would sing the song together, with Mick Jagger only jokingly suggesting that Dylan wrote the song for them. It would be more accurate to say that Dylan knowingly wrote the song for a whole history of musical inspiration that came before.