As so often happens, Dorian ends up being his own worst enemy. The painted Dorian just serves to egg on the flesh-and-blood Dorian, and his fascination with the visible incarnation of his sins is part of what makes him so very, very bad. The portrait serves both as conscience and naughty pleasure – sometimes Dorian is perversely delighted by how disgusting his inner self is, and sometimes it seems to reproach him for his wrongs. Either way, Dorian really pushes himself to be worse with every passing year.
In the simplest of plot terms, James is the most obvious baddie – he wants Dorian dead, period. Purely motivated by revenge (and not exactly by intelligence), James's plotline is a pretty direct one.