Joe is definitely the boss; he's like everyone's father. There's lots of tough love, but that's just because he has everyone's best interests at heart. He wants a successful diamond heist with no drama, so he's assembled this particular crew. He's obviously been in the business a long time, so he's got a large pool of contacts to select from. Joe may be a little forgetful when it comes to remembering names from an address book he hasn't seen in forever.
He's also the man with the plan, and the target of the undercover operation. The police are willing to risk the life of their undercover guy just to nail Joe. He's a major crime boss.
We get to see a human side of Joe every once in a while, just to remind us that this guy is a family man, a man who values community and loyalty. We hear him talking on the phone before Mr. Blonde walks in. The person he's talking to has apparently has fallen on tough times, but Joe isn't worried about the money he owes him. He tells the guy to take his time and keep his chin up. We also see him treat Vic Vega (Blonde) like family. He sends him a lot of packages in prison because Vic was willing to do four years and not squeal.
Of course, Joe's views on loyalty also can work in the other direction. When he finds out Orange is the rat, he wastes no time shooting Orange despite White pointing the gun at him. We like to imagine him thinking it's worth it on his way to the underworld.
To some extent, Joe's a mob-movie stereotype: the violent mobster with a heart of gold when it comes to friends and family, who thinks nothing of going out and blowing people's brains out after he makes dinner and tucks the kids into bed.