Billie Holiday's unforgettable delivery makes the song. The tune doesn't have much of a melody at all, but the way Holiday subtly varies the notes is both haunting and beautiful. In Benny Green's judgment, "Her elocution is superb, with but a hint of a Southern accent; her tone is languorous but unflinching, raw yet smooth, youthful yet worldly." Artie Shaw, who hired Billie to tour with his band for a time, also commented on her dramatic delivery: "She never had any dramatic experience, but there was a sense of inherent drama in her. I don't think she ever took any elocution lessons, but when she said 'bitter' she said it in a way worthy of any Dame on the British stage."
The song's downer energy is probably heightened by the fact that it's in the key of B-flat minor. B-flat minor is a notable choice because it has traditionally been a "dark key"—that is, a key used for sad and gloomy tunes. Chopin's Piano Concerto No. 2 and Tchaikovsky's Marche Slave, for example, are both in B-flat minor. (Incidentally, B-flat has also been proven to really irritate alligators, and it is also the lowest recorded note in the universe. That's right: the sound of gases orbiting a black hole in the Perseus cluster has actually been identified as being the B-flat 57 octaves below middle C on a piano.) But back on subject, the B-flat minor key contributes greatly to the somber tone, even though you're probably unaware of it if you're not familiar with music theory.
Further ruining your mood, the song ends on an F (with Billie's resounding voicing of "crop"), leaving the song somewhat unresolved. It is most standard in Western music for songs to end on the root note of the key that they're in. That would be the B-flat. In his book "Strange Fruit," David Margolick seized upon the significance of this "strangely unresolved note," comparing it to the dangling "dead man on the branch."