We know what time of year the poem takes place from the title. It's mid July. It makes sense, then, that the fruit on the orchard trees won't be ripe yet. So why bring them up? "Unripe fruit" isn't as tasty or even as attractive as brightly colored, ripe fruit. "Unripe fruit" probably has a symbolic significance. After all, the poem is about the passage of time, and the transformation from a young man to the more mature (or "ripe") speaker. Could the "unripe fruit" represent the speaker's past self, the boyish "William"?