If you've read even one page of Ulysses, it's easy to see why Joyce's text would appeal to Derrida. It's all about play at every level. Words spin out of control, and their sounds become more important than what they mean.
In Ulysses, language itself—in all its over-the-top wildness and weirdness—is the focus. Reference gets left in the dust. In other, Derridean words: there really is no outside-text in this text.