TAX COLLECTOR: Our records show you have never paid an income tax.
GRANDPA: That's right
TAX COLLECTOR: Why not?
GRANDPA: I don't believe in it
This is perhaps the most famous scene in the film—and it involves Grandpa hoarding money. Grandpa isn't willing to pay his back taxes, because he says the government does nothing for him, or anyone. One thing the government did during the Depression, though, was pay unemployment, and invest in works programs to get people jobs.
The tax collector doesn't mention any of this though, and so you don't get to hear whether Grandpa thinks it's worth paying tax money for those programs. The truth, though, is that the rabidly anti-tax stance Grandpa takes would make more sense coming from Mr. Kirby; Grandpa is supposed to love community and democracy.
Perhaps that's why Capra undermined the scene at the end, with a throw-off line about how Grandpa doesn't really owe any money. The scene as it stands didn't quite make sense for the film version, so the filmmakers tweaked it.
GRANDPA VANDERHOF: Well now suppose I won't sell them my place, what're they going to do?
NEIGHBOR: That's right, you own your place.
GRANDPA VANDERHOF: Sure I do.
NEIGHBOR: And they're going to need it too, won't they?
GRANDPA VANDERHOF: You bet they will, and it'll take more than money to make me sell my property. Now go on back to work, stop cluttering up the street, we'll all be arrested.
It's true that you can't take it with you, but while you're here, owning your own home can be very useful. Grandpa's (moderate) wealth is a huge boon to his friends. It's odd that later he just sells the house though he knows this will destroy the entire neighborhood. Why didn't he sell it to someone other than Mr. Kirby, if he had to sell? But perhaps Capra didn't think things through; that sometimes happens in the movies.
GRANDPA VANDERHOF: Maybe it'll stop you trying to be so desperate about making more money than you can ever use? You can't take it with you, Mr. Kirby. So what good is it? As near as I can see, the only thing you can take with you is the love of your friends.
That's the title and the theme, right there. Friends, not love. Write it down, live it. And, you know, if things get tough, ask your friends for money (as Grandpa does.)
GRANDPA VANDERHOF: What if all your deals fall through? Might be a good thing for you.
MR. KIRBY: Man, you're crazy.
GRANDPA VANDERHOF Well, maybe I am, but I used to be just like you once. Then one morning, when I was going up in the elevator... it struck me I wasn't having any fun. So I came right down and never went back. Yes, sir. That was thirty-five years ago.
Grandpa was going up in that elevator towards wealth and more wealth… and then he came down, got out of the rat race, and started collecting stamps. The moral is, push the elevator button for love, not money. (Or something like that.)
GRANDPA VANDERHOF: Scum, are we? What makes you think you're such a superior human being? Your money? If you do, you're a dull-witted fool, Mr. Kirby. And a poor one at that. You're poorer than any of these people you call scum, because I'll guarantee at least they've got some friends. While you with your jungle and your long claws, as you call 'em, you'll wind up your miserable existence without anything you can call friend. You may be a high mogul to yourself, Mr. Kirby, but to me you're a failure—failure as a man, failure as a human being, even a failure as a father. When your time comes, I doubt if a single tear will be shed over you. The world will probably cry, "Good riddance." That's a nice prospect, Mr. Kirby. I hope you'll enjoy it. I hope you'll get some comfort out of all this coin you've been sweating over then!
Grandpa gets uncharacteristically riled up and gives Mr. Kirby what for. The "long claws" here reference Social Darwinism—the belief that the rich got rich by being the fittest, and that society is a struggle to see who comes out on top. Grandpa is rejecting that idea; the goal isn't to be the biggest lion in the jungle, but to have the most friends (lion or otherwise.)