O rose thou art sick (1)
While these lines aren't ostensibly about violence, it's hard to read the poem and not think that the rose's sickness is somehow a result of the worm's act of destruction. One gets the sense that the worm's "dark secret love" infects the rose and somehow destroys it, almost as if the rose were pregnant with a destructive disease.
Has found out they bed Of crimson joy (5-6)
Again, this quote isn't overtly about violence, but the sexual undertones of the poem make it difficult not see some kind of violence at work in these lines; sure the worm has only "found out" the rose's bed, but perhaps that is merely a euphemistic cover-up for something more aggressive.
And his dark secret love Does thy life destroy (7-8)
Can destruction be anything but violent? The rose doesn't merely die of an infection; the worm is an active agent of the destruction. The phrase "dark secret love" sounds like some kind of violent sexual act, one that has destructive consequences.