Interestingly, physical appearance is the most commonly referenced mode of characterization here; people are instantly judged based on their looks, and the Governess is quick to make dramatic assumptions due to her perception of people's exteriors. For example, the children are both instantly assumed to be innocent (despite Miles's dodgy past) purely on merit of their angelic appearance, while Quint and Jessel are immediately categorized as "evil" because of their faces and figures. In fact, it's one of very few ways in which we come to understand the people that appear in this story, since we don't see the interior lives of any of the characters except the Governess.
Social rank plays a fundamental role in telling us what to expect of characters. The world of Bly (and of Victorian England) that James depicts for us has a highly structured system of social hierarchy, and any deviation from this system instantly marks characters as bad. Quint, for example, refuses to stay within the acceptable bounds of his social position as a servant, which is what truly unnerves the characters around him. Mrs. Grose constantly comments on how Quint was "too free" (6.7) with everyone, and "did what he wished" (7.20); his transgression of the accepted boundaries between servant and master mark him as a dangerous character. On the other hand, the placid and gentle Mrs. Grose, who questions the Governess's notions but never questions her social superiority, is the image of the ideal servant, who never steps outside her bounds, and is therefore a positive character.