When John and Aleyn arrive at Symkyn's mill and propose to watch their corn being ground into flour, Symkyn immediately recognizes their intention and takes it as a challenge. For him, the conflict is not just between himself and these particular clerks, but one between all clerks and the working-class people he represents. Symkyn sets out to cheat Aleyn and John just to prove that clerks aren't as smart as they think. The miller thinks that the tricky arguments that scholars are prone to are much more dishonest than him tricking people out of a little corn. In his mind, the theft of these scholars' corn becomes tit-for-tat, trickery for trickery.
Although Aleyn and John have right on their side, they, too, view their ability to prevent Symkyn from tricking them as indicative of their superior brainpower. When he manages to steal from them anyway, John worries that everyone will call him a fool. What might have been a simple business transaction has, through the hands of these hyper-competitive types, become a conflict between the scholarly-smarts of the clerks and the street-smarts of the miller, with no clear winner at the end of the day.
Symkyn's reaction to the clerks' announcement that they will watch their corn being ground turns what might have been an ordinary business transaction into a competition between the scholarly and the working classes.
Symkyn correctly interprets the clerks' opinion of him as a big dumb oaf.
Symkyn incorrectly interprets the clerks' opinion of him; their actions at his mill do not stem from an opinion of him as being a big dumb oaf.