From the very beginning, the song places us in Nazareth, a town with mega-historical significance. It consequently seems fitting that the devil, (Miss) Moses, and Luke should be wandering the streets of Jesus' childhood home.
But according to songwriter Robbie Robertson, this is Nazareth, Pennsylvania, not Nazareth, Israel. It's the home of Martin Guitar, not Christianity's holy family.
German guitar maker C.F. Martin immigrated to America in 1833, briefly settling in New York before setting up shop on a 55-acre spread outside Nazareth, PA. Today the company is run by C.F. Martin IV and is still based in Nazareth. Eric Clapton, Stephen Stills, Johnny Cash, and Joan Baez have all played Martin Guitars.
There is something legendary, then, about the Nazareth of "The Weight," but it's not the legend we thought it was at first glance. Other Band members add to this mind-tweaking misdirection by telling us that most of the characters in the song are not who we probably think they are. Luke was former Hawks' guitarist Jimmy Ray Luke Paulman, and Carmen, the Devil, and Miss Moses were well-known, if not legendary, characters from Arkansas.
So the setting throws us for a bit of a loop, but it also encourages us to think about legend and mythology. People and places acquire mythological importance over time; they don't start out that way. Before they become the snarl-toothed prophets of the Old Testament, they're just Crazy Chesters rambling the streets of an Arkansas town. Perhaps what this song is trying to say, then, is that the grand experiences that acquire mythological status are, in the beginning, just the universal trials of common people.