We're talking about the unnamed lady's principles here, not the speaker's. We're guessing – partly based on the poem itself, and partly based on Byron's reputation – that, given the opportunity, he'd happily seduce her. But this particular woman would have none of that. We're told repeatedly that she's pure and innocent, and that's part of why she's so gorgeous.
The speaker opens the poem with a description of the woman's beauty, and ends it with praise of her virtuous "innocen[ce]" in order to emphasize that true beauty requires a balance of good principles as well as physical beauty.
The speaker's emphasis on the woman's principles, supposedly in praise of her beauty, suggests that he wishes to dodge suspicions that his attraction to her is less than innocent.