Shortly after his heart-to-heart with Marius, Valjean visits the Gillenormand house and meets with Cosette. He basically tells her that she shouldn't call him "father" anymore, since she has a husband and won't need a father. Cosette thinks this is silly, but Valjean insists.
As time goes on, Jean Valjean periodically leaves his house and walks toward Cosette's new home. But he always stops midway in his walk and turn back home. It's all pretty heartbreaking.