Does the speaker literally mean he was the color "green"? Let's hope not, unless he was an alien baby. Green here probably metaphorically represents being new and fresh and full of life. And also a total noob. After all, if you're green, it means you have absolutely no clue what you're doing.
And nothing I cared, at my sky blue trades, that time allows In all his tuneful turning so few and such morning songs (42-43)
We're nearing the end of the poem here, and the speaker's tone has shifted. Instead of being young and carefree, he says, "nothing I cared." What's changed? It sounds like rather than celebrating being carefree, he's making a confession, admitting, that, yes, in retrospect, when he was young, he didn't have any cares and didn't realize things were going to change.
Nothing I cared, in the lamb white days, that time would take me (46)
This is a repetition of the sentiment stated in the quote above. The speaker repeats "And nothing I cared." You could say that repeating the same phrase alters the meaning a bit. The first time he says it, he may be just admitting that he was being careless when he was younger and living footloose and fancy free, unaware that things were going to change. But repeating that phrase makes it sound insistent, possibly frustrated. The speaker could be saying, with a twinge of regret, that he didn't care time was passing and didn't realize his blissful childhood would end so soon.