Your argument basically breaks down to “If we allow A then we must allow B” which makes the assumption that A=B where A is stopping Nazi hate speech and B is stopping antifa speech. Ignoring that Antifa isn’t holding rallies or making speeches there is still the simple problem that A =/= B. We cannot logically equate Nazi hate speech with those who fight for freedom. This should be obvious to anyone and why I have no fear the criminalizing Nazi speech specifically would lead to the slippery slope you hand wring about.
Indeed, we have legal precedent to suppress certain kinds of speech while still protecting the freedoms of the people. The low hanging fruit is yelling fire in a theater. This is prohibited because a reasonable person would see yelling fire as risking the lives and well being others. Similarly, a reasonable person can see that calling for racial violence, national socialism, and the subjugation and elimination of anyone who does not fit the Nazi ideal as being a threat to the safety and well being of others. When we pretend this is not the case, arguments such as yours may sound reasonable. They are not. Nazism represents a clear and present danger to the United States, it’s people, it’s government, and indeed the world. I cannot see how anyone might argue otherwise. So, it is with this in mind that we may move forward and call for the criminalization of Nazi speech and activity.
Finally, I’d like to point out that by skirting the law, many Nazis these days are able to avoid arrest by making sure they do not target a specific person or commit any violence themselves. Instead, they encourage violence from their followers leaving them in the legal sweet spot. This is terrorism by proxy. It is a way to abuse the freedoms which a nation may protect while behaving in a manner that would otherwise be criminal by using a loophole in our laws which clearly needs mending. Without mending this loophole, the real possibility exists that these people may flourish and that our nation will suffer for it.