Asher has a Harry Potter-Draco Malfoy relationship with his yeshiva classmates in that he's Harry and his classmates are pretty much all Draco. They tease and taunt him because he's got a gift that sets him apart:
"The thin pimply-faced boy who sat in front of me stood in the doorway to the classroom with a group of four or five other students. He watched me come up the corridor.
'Here comes the desecrator of sacred books,' he said in Yiddish in his loud nasal voice.
The ones near him laughed." (142.2-3)
Asher has pretty much no friends in his yeshiva, which makes his life there very lonely. The loneliness and isolation are both factors which contribute to his estrangement from his religion and make him retreat deeper into his art.