Poor old dad. He watched his son go to prison, go blind, be betrayed by both his wives, and now, when he finally thinks he has a plan for how to bring Samson home, Samson refuses.
We definitely see the most hopeless and despairing side of Samson when he's talking to his father. Unlike his other single visitors, Dalila and Harapha, who are on the side of the Philistines, Manoa is completely on Samson's side. Obviously—they're family. But Samson seems to be so depressed that even this family visit can't cheer him up.
To be fair, Manoa doesn't seem to understand the inner aspect of Samson's suffering. He seems to think that Samson is bummed about prison: "Thou must not in the mean while here forgot/ Lie in this miserable loathsome plight/ Neglected" (479-81). No such thing. Samson tries to explain that he's depressed because God has abandoned him and let down his people, but Manoa doesn't even seem to hear him. So we're left to wonder if Manoa's motivations are actually a little selfish. Does he care about Samson at all?
It's also worth noting that Manoa's attempts to have Samson released often sound like they're going to involve a little under-the-table pay-offs to the Philistines, suggesting that Manoa doesn't understand that Samson doesn't want to make anymore agreements with the Philistines. As far as Samson's concerned, been there, done that.
Dealing with the Philistines would be, for Samson, worse than staying in prison. So Manoa, no matter how good and political he seems, actually turns out to be the opposite: he's a weak opportunist who's willing to compromise his people, principles, and religion to secure a personal benefit. No thank you.