Captain Smollett is a stand-up guy, but he's also all about the discipline. The way he knows there's something wrong with the crew is that they've been spoiled: Squire Trelawney has given them a barrel of apples and regular rounds of alcohol, total proof that this is a bad crew. And yeah, Captain Smollett is proven right: the crew is spoiled and their lack of discipline is proof that this is a pirate crew. At the same time, Captain Smollett's discipline, even though it is well-meant, also prevents creativity and invention. Captain Smollett's hard line with Long John Silver is totally at odds with Doctor Livesey's later, more flexible (and successful) negotiations.
What's more, even Doctor Livesey comments that, if Captain Smollett had not been injured in their first battle with the pirates, Jim would not have dared to leave his post at the fort. Now Jim's actions may have been questionable, but we can't deny that Jim's departure allowed him to steal the Hispaniola from the pirates, which was a huge blow to their morale. So while Captain Smollett's discipline and attention to duty is an important balance to Squire Trelawney's recklessness, his insistence on following the rules at all times prevents him from seeing the value of practical and cunning plans.
A great example of Captain Smollett's rigidness against his own interests is when he insists on flying the British flag over their fort in the forest. Squire Trelawney points out that the flag is giving the pirates a target for their cannon, but Captain Smollett absolutely refuses to lower the flag. Sure this is noble and patriotic and all – and the other men in the fort are impressed by his bravery – but we feel that, when there are cannons going overhead, it's kind of dumb to insist on rules and regulations at all costs.