The poem concludes by telling us that the worm "destroys" the rose's "life." The worm's weapon of choice is his "dark secret love," a phrase that implies some kind of violent sexual act, especially when we recall that the worm first has to penetrate the rose's "bed of crimson joy." There is no natural reciprocity or symbiosis in the poem's vision of nature; the only way the worm can interact with the rose is through violence.
The worm isn't really violent; he "destroys" the rose, sure, but to call it an act of violence is to read something into nature that isn't there.
The violence in this poem isn't as bad as we might think; it's part of the natural cycle of life and death, of the food chain.