There are two kinds of inventions that fuel progress. Technological inventions like your smart phone or a bullet train give us what past generations would have thought of as super powers. Social inventions like nation-states or the modern corporation also transform lives.
If people mattered more than inventions, a "good enough" person could run faster than a bullet train or a "good enough" person could create a great and prosperous life even in the midst of anarchy and chaos. In the words of W. Edwards Deming, though, "A bad system will beat a good person every time."
Racist: my race is better
Patriarch: men are better
The argument of racists and patriarchs is that we white men have ended up at the top of the heap because we're better. The simplest evidence of this is that we're wealthier and more powerful than people of color or white women. It is not that we've had better access to better inventions. It's simply that we're better people.
But differences in people do not explain differences in lifestyle or wealth. An American in 2000 made 6 to 8X what an American in 1900 made and could expect to live 30 years longer. Not because the American in 2000 worked harder or was morally better or was innately more intelligent. The average work week in 1900 was sixty hours (10 hours a day, six days a week). The average work week now is less than 40 hours.
It is true that Americans in 2000 were "smarter" than Americans in 1900 but that was not because of any innate intelligence. It was because of the popularization of social inventions like a K-12 education and the modern corporation that our efforts were leveraged into such (relatively) great lives. The American in 2000 simply had far better inventions than a person in 1900.
The notion behind white supremacy or anti-immigration, or patriarchy is that people matter most. In the minds of these people the systems are invisible and the people who sit atop them is all they can see. It's like thinking that the driver of the car is responsible for going 70 mph rather than realizing that anyone in that position could match or better the speed.
There are a host of reasons why racists and patriarchs are such passionate defenders of the border. A chief one is that they're convinced that the people living in more impoverished parts of the world are that way because they are lesser people. In fact, what has made our country great is that the inventions we've popularized - from Montessori schools to smart phones - make us better. It doesn't just work for our children; it can work for anyone's children.
When you welcome immigrants you're expressing confidence in the power of your systems to change life for anyone. When you reject immigrants you're saying that our inventions - our systems - are incidental to who we are and if other people come here to use them those people will still be lesser people, will be a burden on us.
There are obviously limits to the number of immigrants a country can absorb and migration laws do matter. But focusing on the border as a basis for prosperity is misleading for so many reasons. The most important reason that it shifts a country's focus and policies onto protecting borders rather than creating inclusive inventions that make more people more able to make themselves and others happy. It shifts the focus from inventing and change to protection and resistance.
We've gotten off the path to progress. Hopefully someone will still find that path or else the global economy will stagnate and when modern economies stagnate, modern democracies flounder.