In a previous article, I have already tackled the fundamental notions of frequentist statistics as violating the desiderata of science. Be that as it may, the stupendous extent of this approach’s contradiction with its own norms merits some attention. This is Ioannidis’ main argument, as also John Cook (see Cook’s blog post).
To quote Cook,
Here’s an example that shows how p-values can be misleading. Suppose you have 1,000 totally ineffective drugs to test. About 1 out of every 20 trials will produce a p-value of 0.05 or smaller by chance, so about 50 trials out of the 1,000 will have a “significant” result, and only those studies will publish their results. The error rate in the lab was indeed 5%, but the error rate in the literature coming out of the lab is 100 percent!
I may point out to the discerning reader that even if our frequentist scientists started reporting all their non p-value results, the 5% error rate conclusion is still based on a fundamental misunderstanding of probability that prevents us from using the results of such experiments in rational decision-making.