What I wrote is also about something people sometimes say in certain contexts, not about whether they “are racists.”
As for defending free speech rights, you have plenty of room to do that. If that’s what you want to do, go right ahead. Whatever I post can’t stop you.