In Book 3, we get introduced to the Marauders, that delightful gang made up of Sirius, James, Remus, and that traitor Wormtail. It seems appropriate that Fred and George are holding on to the Marauders' Map of Hogwarts, because the Marauders have always reminded us of Fred and George: Gryffindor's major pranksters, but basically good-hearted. However, it turns out that Fred and George's pranks aren't always so lighthearted – Montague is still recovering from his Vanishing Cupboard adventure at the end of the book. And the Marauders also have a much darker side than the fun boyhood memories Sirius passes on to Harry.
When Harry looks into Professor Snape's Pensieve, he sees his father, James. And James is constantly showing off with a stolen golden Snitch and artfully tousling his hair. He seems utterly vain. When he and Sirius spot Severus Snape sitting quietly, they decide to ambush him out of sheer boredom. So James is not only arrogant; he is also a bully.
After seeing his father in Professor Snape's memories, Harry has a hard time reconciling this preening jerk with his ideal image of James Potter as a good father and brave member of the Order of the Phoenix. He goes to Sirius and Remus to ask them about James. Both of them agree that, yes, James was arrogant as all get-out at fifteen. Still, he improved. He managed to start dating Lily at the end of seventh year, "once James had deflated his head a bit" (29.210). So Harry has seen his father at his worst, but that is in no way the whole picture of who he is.
The great tragedy that Harry has to struggle with is that he will never see his parents as the entirety of what they were. All he has to go on is opinion and other people's memories. And when opinions conflict, he has to make his own decisions about who to trust. He seems to be getting to know his parents through process of deduction, since he obviously can't meet them face-to-face. For now, Harry is not sure what he thinks of James Potter, but we're sure there's more out there for Harry to discover.