Most of Invisible Man takes place in the narrator's memory, which inherently brings up issues of how well memory works – in other words, the nameless narrator character is choosing specific scenes to portray in specific ways; the entire novel is written from his perspective. Aside from this observation, memory and the past are also important in the novel as the narrator reflects on his past and uses the perspective to derive new feelings and opinions on his experiences. Although the Brotherhood tells him to put aside his past, we see that the narrator's personal journey requires him to square with his past, to acknowledge and embrace it.
Because one's past helps comprise one's identity as a person, the narrator learns to embrace it rather than renounce its perceived shameful parts, as he was originally taught to do.