I scarce believed thou couldst be gone away ; (line 2)
The speaker goes through distinct stages of coping with his lover's absence, beginning with disbelief. It takes him twenty years to figure out that she has gone away. But the key question is: where has she gone?
fed on favours past (line 3)
To "favor" someone is an old-fashioned way of saying that you dig them. At one point, this mysterious woman apparently did have strong feelings for the speaker, or at least she led him to think so. He relives their relationship in memory.
And forty on hopes that thou wouldst they might last ; (line 4)
The third stage of coping for the speaker is hope. He hopes that she will continue to "favor" him. To quote a famous pop song, his attitude is: "I want you to want me."