No doubt about it. If on a winter's night a traveler is not your normal, straightforward story. It might take a good long pause to step back and realize what Calvino is doing—jumping from second-person narration into the first chapters of novels that don't actually exist. Throw in his tendency to go off on philosophical rants, and you've got a book that's going to toss you a few tough curveballs.
Once you become comfortable with identifying yourself as the Reader, though, it can all start to come together. You still might not understand everything the book is talking about; but worry not. Some of it is meant not to be understood.