Sure this is a thing. But it gets a little queasy when you go too far down that line of reasoning. “He’s ugly so what right does he have to criticize how someone else looks” seems to covertly signal the idea that someone who is not ugly does have that right.
