Maybe, maybe not. The first wave of fear-mongers was aimed at working class brits, because the rich, white people were scared of them. As you note, the “wrong people having babies” has generally been at the roots of these scares. It was at the root of the eugenics movement, and we see where that has gotten us in the 20th century. Mass murder became an elitist past time.
Part of the problem was almost always related to access to resources, and who gets them. There is plenty of evidence to suggest that various 19th century “famines” were pretty much just down to colonial policy (see Mike Davis’ Late Victorian Holocaust, where he lays out this claim pretty convincingly, as well as much of the scholarship on the Irish famine of the 1840s, where the British were moving food OUT of Ireland, while the western peasantry starved).
I think that depends on what we do to prepare for it, instead of running around worrying about illegal immigrants, national borders, and making sure everyone is “pure” enough to exist. Until we let go of this insane need to dominate resources via a capitalist economy, and instead use what we’ve learned from the capitalist period to make sure we all have enough, we’re sunk.
