When Frodo and Sam first meet Faramir, he and his men are on a mission in Ithilien to fight back the Southrons, also known as the Swertings, also known as the Haradrim. Phew. These are evil men who have come from the South to help Sauron fight Gondor. Faramir's men make reference to the fact that Gondor and the Southrons were once allies, but that was a long time ago. They have been enemies for many generations. What we find interesting about the Southrons is that they do seem kind of racially marked. They ride elephants (well, Oliphaunts) and have black hair and black eyes. At any rate, they are different from the Riders of Rohan and the men of Gondor. Consider Gollum's description of the Southrons:
We have not seen Men like these before, no, Sméagol has not. They are fierce. They have black eyes, and long black hair, and gold rings in their ears; yes, lots of beautiful gold. And some have red paint on their cheeks, and red cloaks; and their flags are red, and the tips of their spears; and they have round shields, yellow and black with big spikes. Not nice; very cruel wicked Men they look. Almost as bad as Orcs, and much bigger. (4.3.68)
We definitely do not want to suggest that the Southrons represent a particular race or nationality. But it is enough that they are Other than the people of Gondor. Their evil appearance is partly a result of the fact that they are unfamiliar and different from Faramir's people. Tolkien relies on physical features to make that otherness clear.
To be fair, Sam can look past this difference in appearance to see the Southrons' common humanity. When Sam observes a fallen Southron soldier, he wonders if the soldier wanted to be here going to war with Gondor. Was he forced into it due to his circumstances?
It seems like a waste to Sam to be spilling all of this human blood, both Gondorian and Southron. Sam's reminder that the Southrons are human, too—whether they are "very cruel wicked Men" or not—shows us that men (maybe unlike orcs) are not by nature good or evil. We have a lot of different reasons for fighting, and it doesn't make any of us any less human.