Critics loved it too and despite the dissenting voices condemning its perceived immorality, it received ten Oscar nominations and five wins, Best Picture, Best Director, and Best Original Screenplay among them (all three going to Wilder himself).
There may well have been some misogyny in the pastor's glancing blow, but the real issue for him was the perceived sexual immorality in all of the women's stories.