A threshing floor is a structure that's used to separate the grain from the chaff of wheat (the yummy parts from the inedible bits). Wait, what does this have to do with Harlem? Not much, except that it's an important image from the Bible, which is a big deal in this Harlem church.
In the Bible (Matthew 3:12, to be exact), the threshing floor is used as a metaphor for God's judgment:
His winnowing fork is in his hand, and he will clear his threshing floor, gathering his wheat into the barn and burning up the chaff with unquenchable fire.
So the idea is that God will take the good (the wheat) up to heaven (the barn) and burn the bad (the chaff) in hell. Yikes.
The image first shows up in the novel when Gabriel is lamenting his sons' evil ways: "Neither of his sons was here tonight, had ever cried on the threshing-floor" (2.2.105). "Here" is the church, and the idea is that neither of his sons have ever cried and been saved; their evil parts have never been separated and thrown away.
The entire third part of the novel is entitled "The Threshing-Floor" and it's all about John's conversion experience. He finds himself lying on the floor, rolling around in the dust, and when he's fought all of his demons off he comes out of the trance a saved man.