I agree that it is pretty easy to attack someone with a charge of racism. Usually it’s better to say that someone said something racist or did something racist than that they are racist. So maybe we should say that Mr. Cohen’s column is racist rather than that Mr. Cohen is racist.
But I don’t think leaving racism out of the picture does justice to the situation. Mr. Cohen’s comment supports racist sentiment by denying that a group of people are racist with one breath and then suggesting that they would gag at an inter-racial marriage in the next. If someone who gags an inter-racial marriage isn’t necessarily a racist, we could certainly say that they display a very racist impulse.
To compare inter-racial marriage to inter-religious marriage or inter-generational marriage exposes why it is problematic. If you ask those people who quit your friend why they did it, they would likely tell you that Jehova’s Witnesses are nuts and it was crazy of him to take up with one. If you ask people why they think inter-generational marriage is wrong they will tell you that those marriages seem exploitative. Maybe people are have a prejudice against Jehova’s Witnesses and maybe people have strong age-based biases as well, but that is precisely the point! If a person thinks that black people shouldn’t marry white people, it is probably because they have a bias against either black or white people.