I believe it to be based on the idea of “if we restrict the speech of Nazis while we’re in power, they’ll restrict our speech when they’re in power”.
Guess what? Empowered Nazis are going to restrict your speech no matter what you do. If you want to avoid that, you have to avoid giving power to Nazis.