It’s called a ‘familywise error rate’ and, yeah, correcting for a number of tests is an ancient statistics trick. I’ve used, ooh, at least half-a-dozen various corrective measures to deal with it when doing post hoc tests. It’s basic statistics.
However, the issue isn’t just the number of tests, it’s the sample size. A huge sample size covereth a multitude of sins, and if you get it stupidly huge enough you can produce results with almost arbitrarily small p-values even after the most stringent corrections.
And these results won’t be the result of chance necessarily, but they still might be spurious. A statistical method is, in many ways, like picky filter amplifying one particular signal. However, like with any amplifier, if you crank it up too high the noise will get amplified alongside the signal. Thus, a lot of results are absolutely true and correct and perfect in every way except they don’t show real-world effects but, instead, detect with magnificent precision defects in the experimental protocol or statistical method used.
There are ways to handle this, incidentally and they aren’t even that complicated. You use multiple approaches at the same time that share as little of the underlying structure as possible to make sure your statistical methods aren’t screwing with you. You always always cross-validate to correct for overfitting on the available data-set[1], you run all your experiments triple-blind to avoid any sign of bias, and once you’ve crossed all the t’s and dotted all the i’s you then hand the whole mess to a completely different research group to replicate, ideally using a completely different data set.
It’s a lot more work, however, and costs a lot more and nobody is going to do it much because the present system encourages opposite behavior. For the industry either the company’s worth is predicated on producing seemingly impressive results as quickly and cheaply as possible[2] then that’s what you have to do or be outcompeted by someone who will or you’ve commissioned the research and the company doing the actual work is, again, incentivized to get results as cheaply as possible.
In academia the problem is the publishing rat-race. You need that publication, badly, and no journal is going to publish your impeccably researched null result. But they might just publish your very-marginal-if-you-don’t-look-at-this-one-test-we’ll-not-do research since it produces a result.
I do a hell of a lot of peer review and if I had a dollar for every bit of mathematical legerdemain I’ve had to call authors out on I could… well okay, a dollar isn’t that much, but I could afford a damned fine bottle of whisky.
The industry stuff you can’t fix: that’s your basic flaws-of-capitalism stuff. The academic stuff you could fix by requiring that all studies be pre-registered and for according full credit for replications, but what sort of institution could make such a sweeping change stick I can’t imagine.
[1] In fact, you run multiple-ply crossvalidations for added security. Mind, that gets expensive when you have to run your analyses on supercomputers but that’s just how the cookie crumbles.
[2] And as an occasional statistician, I assure you, the only thing stopping me from pulling rabbits out of my hat on command is professional integrity and honesty.
