partly because of my love for you, partly because of your love for yoghurt (4)
This is a pretty casual expression of love, don't you think? Usually there's a choir in the background, some gently lit snowflakes, or, heck, even a lone violin. But in this case we get… yoghurt? This gives us an idea of the casual, conversational tone of the poem to follow. The speaker is just totally head-over-heels for the you, but he's also trying to avoid going too far over the top. He follows up his declaration of love—a potentially vulnerable moment—with an attempt to equate it with the addressee's love of yoghurt. This lets us know that we're not in for your typical lovey-dovey romance.
partly because of the secrecy our smiles take on before people and statuary (6)
One of the great things about love is the privacy it creates. When you're in love with someone, it's like you live in a world with just the two of you. You have all these inside jokes and secret looks, and they are only meant for that other person. We get the sense here that the speaker and the "you" of the poem are a kind of exclusive club (with lots of velvet ropes and really big doorman to keep out the rest of the world).
in the warm New York 4 o'clock light we are drifting back and forth (9)
This sounds a little trippy. The speaker's not talking about astral projection, though. He's talking about that floating-on-a-cloud feeling that comes when you're in love. Notice, too, that the couple is "drifting back and forth." They are leaving themselves (though not literally, we hope) and moving toward each other. You know that stage of love where you sort of… become the person you are in love with? Or at least feel closer to them than you thought possible? This is what our speaker is on about here.