Conservatives expect things to get worse*, and the absence of actual immigrants tends to be associated with greater anti-immigrant tendencies. So it won’t happen.
*The classical expression of conservatism in economics is Hutber’s Law: Improvement causes deterioration.