Pip is off immediately, but he decides to stay at the village inn rather than Joe's house because you just know that Joe is going to tell him that his high school curfew is in effect and he has to do his chores.
The journey home is pretty much the carriage ride from hell. There are two convicts who accompany the carriage riders, and—what do you know?—one of the convicts is the very same man who gave him the two pound note all those years ago at the Three Jolly Bargemen.
Pip has to sit right in front of this convict, and, as he's a heavy breather, Pip feels his convict-breath on him the whole ride home.
The convict tells his compadre the story of giving a two pound note to a little boy in marsh country many years ago, and Pip realizes that he's relating the exact same story.
He's freaked out that the convict is going to recognize him, even though that's highly unlikely, Pip jumps ship (or carriage, rather) at the first possible stop.
When he gets to the Blue Boar, everyone there recognizes him as the young man for whom Pumblechook is responsible. We say again: gross.