Think of all the time you are not Wasting, and would not care to waste,
So the speaker thinks that anything other than what they're doing would be a waste of time. In other words, all the things he listed before these lines—the reading, the gardening, having lunch with friends, listening to a college lecture—are all huge wastes of time compared to what they're doing now.
But what are they're up to? We still don't know exactly.
It's interesting that our speaker is still asking her to think about all those things she could be doing. Maybe he wants her to value the moment the same way that he is.
Such things, thank God, not being to your taste. Think what a lot
The "things" he's referring to are the long list of activities he listed in the first half of the poem.
So the things he listed, apparently, aren't really activities the woman likes to do. Well, that means she's not much of a reader, shopper, gardener, or dog person.
"Thank God" reinforces how grateful the speaker is that the woman is not doing anything else, except being with him.
This is sounding more and more like a romance—the speaker is super-thankful that she's with him and not doing other things.