I scarce believed thou couldst be gone away ; (line 2)
The speaker goes through distinct stages of coping with his lover's absence, beginning with disbelief. It takes him twenty years to figure out that she has gone away. But the key question is: where has she gone?
fed on favours past (line 3)
To "favor" someone is an old-fashioned way of saying that you dig them. At one point, this mysterious woman apparently did have strong feelings for the speaker, or at least she led him to think so. He relives their relationship in memory.
And forty on hopes that thou wouldst they might last ; (line 4)
The third stage of coping for the speaker is hope. He hopes that she will continue to "favor" him. To quote a famous pop song, his attitude is: "I want you to want me."
all being one thought of you ; (line 7)
The speaker eventually merges himself completely into one big ball of thought about the woman. Love is often thought to be the merging of two souls, and the speaker reenacts this process in his mind.
Or in a thousand more, forgot that too. (line 8)
Wait, so let us get this straight: the speaker has become so wrapped up in mourning his lover's absence that he…forgets about his lover? Maybe the point is that his mind has become a total blank, an emptiness, because she is not there to fill it.