Sometimes perhaps you don't want to be a part of me. Nor do I often want to be a part of you.But we are, that's true! (34-36)
After saying this whole time that it's not simple or easy to know what's true, the speaker is finally happy to come across something that he says in true. In fact, he's so happy that he even expresses it using an exclamation mark—woo-hoo, that's true! But the truth here isn't exactly gleeful or happy. The truth he finds is that even if the two races hardly get along at all, they're stuck being a part of each other.
although you're older—and white— and somewhat more free. (39-40)
Perhaps it's hard for the speaker to know what's true because he lives in a country that was founded on principles of equality and opportunity, but allowed slavery, and still, at the time of this poem, allowed systematic discrimination. What does it mean for truth when one person is freer than another?