Add giants to the list of people Milton doesn't like, because Harapha definitely comes off the worst. Unlike Manoa and Dalila, who both appeal to Samson to let them take care of him. Harapha's just there for the show: "Much have I heard/ Of thy prodigious might and feats perform'd... And now am come to see of whom such noise/ Hath walk'd about" (1083-1089).
This language of sight and spectacle not only foreshadows the upcoming public officer who will order Samson to come literally put on a show for the Philistines, but also reminds us of the contrast between Harapha's ability to see and Samson's blindness. As another big guy famous for being super strong, it's pretty obvious that we're meant to see Samson and Harapha as counterparts or foils (see "Character Roles" for more about this): similar in lots of obvious ways, different in some crucial ones.
One way to think about their difference might be in terms of time. Harapha, all proud and strong and unimpaired, is kind of like what Samson used to be. He represents Samson's past. In fact, we could go even a bit further and think about Harapha as a version of Samson and their encounter as a confrontation between Samson and himself.
If this is starting to sound familiar, it's because every single one of Samson's visitors reflects back some aspect of Samson's own character that he then has to face and grapple with. So even though Harapha refuses Samson's challenge to physical combat, he has inadvertently engaged Samson in some serious intellectual combat.