Power
As an individual, you don't have much power yourself. It's not like you're the President of the United States, telling the crew of Air Force One exactly where you want it parked. You carry and transport bags, remember? You're not even as powerful as the person waving those glowing orange cones around (also known as an aircraft marshaller)—and they're basically just being paid to dance.
You have a bossy supervisor to listen to and a pushy airline making sure you don't lose or misplace bags. The pilots and flight attendants may be friendly in general, but some might be full of themselves and decide they have no time to associate with the likes of you.
On the bright side, you'll almost never have to deal with a customer while on the job, because even if you're not throwing bags around on the runway, you've at least got those massive headphones you can wear to block them out.
One source of true-blue power that you have isn't as one individual baggage handler, but as part of a group of baggage handlers (which one might refer to as a baggle*). As one of many, many unionized baggage handlers at a specific airline, you have some power to tell the company how much you think you should be making.
On occasion, irate baggage handlers have been known to stage strikes and walkouts. And there's not much an airline can do except re-book passengers' flights, and either negotiate with the handlers or bring in "scabs" to do their work. So you've got some power, but only when things get really bad.
*That's a joke, not a fact.