Like Ashlee Simpson and Lana Del Rey, the Rabbi is someone the Levs love to hate. He's a very controversial figure in the lives of Asher and his family, to say the least. Asher is terrified of him because he's an authority figure who might disapprove of his art. Rivkeh is furious with him because it was while traveling for him that he brother died. And Aryeh is strongly devoted to him. Yeah, this isn't going to turn out well.
The Rabbi himself tries to lead his congregation in the kindest and most judicious manner possible, which means that he tries to be understanding of Asher and even arranges for Asher to work with Jacob Kahn. But after Brooklyn Crucifixion I and II, he feels he has no choice but to send Asher away:
'I do not hold with those who believe that all painting is from the sitra achra. I believe such gifts are from the Master of the Universe. But they have to be used wisely, Asher. What you have done has caused harm. People are angry. They ask questions, and I have no answer to give them that they will understand. Your naked women were a great difficulty for me, Asher. But this is an impossibility…I will ask you not to continue living here, Asher. I will ask you to go away.' (366.4)
The Rabbi tries his best to incorporate Asher into the Hasidic community, but he ends up struggling and ultimately losing his grasp on the important work Asher is doing, hence his decision to banish a talented artist from his hometown.