How we cite our quotes:
So are you to my thoughts as food to life,
Or as sweet seasoned show'rs are to the ground;
And for the peace of you I hold such strife,
As 'twixt a miser and his wealth is found; (1-4)
The transition from lines 1-2 to lines 3-4 sums up the central fact about the suffering experienced by the speaker of this poem: it's the flipside of what's pleasurable. As you can see, the first two lines here describe nice, pleasant, beautiful things. But because those things are so nice, people can experience an overpowering desire for them—which can sometimes prevent them from ever being satisfied. This is the key to the suffering (or, as he puts, it "strife") experienced by the poem's speaker. Just like a miser, he can never truly enjoy the thing he loves most.
Now proud as an enjoyer, and anon
Doubting the filching age will steal his treasure; (5-6)
These lines show how the speaker's suffering is typically experienced: as a kind of conflict within himself that makes him unable to settle down with what he's got. One moment he's proud with what he's got, the next he's afraid that other people will steal it. The poem as a whole makes it seem likely that this process doesn't just go from A to B, but that, instead, the two possibilities constantly flip-flop back and forth from one to the other.
Now counting best to be with you alone,
Then bettered that the world may see my pleasure; (7-8)
Here we see pretty much the same idea as in lines 5-6. Here, though, we get more of the sense that the speaker is the cause of his own misery. If it's the speaker's own desire to showcase his love to the world, he can't really blame the world for showing an interest, can he? Unfortunately, that's just too reasonable. The speaker will probably have to hit the rock-bottom of despair before he gets off the painful course he's on.