How is the Hebrew word for “nation” derogatory? It has no innate derogatory meaning, unlike, for example, shiksa or sheygetz (which are extremely disparaging words for non-Jewish women and men, respectively). Certainly it can be used in derogatory ways, but I would argue that that’s no different than any other minority’s exasperation and exhaustion when dealing with a majority.
Or, TL;DR: the only difference between a shout of “Gah! MEN!” from an irked woman and a shout of “Gah! Goyim!” from an irked Jew is the specific language being used.