I had authored an article in 2011 titled “A Critique of Statistical Hypothesis Testing (SHT) in Clinical Research.” This paper highlights philosophical and practical problems that make SHT less than scientific and more than problematic. I shared this recently with a scientist friend, who wrote back:
“Remember, statistical testing is even in vogue in particle physics, because no experiment is without experimental errors, and error estimation requires statistics. Decision Theory cannot base itself on Black and White Information. It is always a tinge of grey!!! Does this consideration make any difference?”
True (that SHT is in vogue in particle physics). That does not imply that frequentist statistics makes sense. Decision Analysis is largely based on Bayesian probability theory, so it is designed from the ground up to deal with uncertainty (or many shades of grey).
It makes our consideration of probability even more important. I might add that my teacher might be Prof. Ron Howard, but his teacher in Bayesian foundations was the great physicist, E. T. Jaynes, who wrote (but did not finish in his lifetime) the seminal book, Probability Theory: The Logic of Science. His notes have spread far and wide, and the book was finally completed by a student.
The problem is that most frequentist physicists nowadays actually do not understand probability theory. If you ask them what it means, they will likely say something like the frequency to be discovered in nature. The notion that two people observing the same event might have different probabilities does not exist in their world, and if admitted, brings down the entire frequentist science.
In the Bayesian worldview, the key question is, “What did you believe before you did the experiment?” You can only learn based on what you knew, and what you believed you could possibly learn. Error cannot be properly defined without the prior probability and the likelihood. To expand a little bit, I can believe that there is a 40% chance of rain tomorrow. Then, I can consider a rain detector, and characterize its sensitivity (chance of true positive) and specificity (chance of true negative). By doing so, I have a complete model for the detector’s accuracy — given the experiment’s result, I know how to change the probability of “Rain.” Can you explain how to update the probability of Rain based on experimental results using the frequentist method (p-tests, etc.)?
The frequentist physicist’s probability is not a probability — it is something weird that cannot be used to make decisions, handle uncertainty logically or support the updating of our beliefs. There is no true probability distribution in the universe. Probability is an expression of our belief/knowledge, existing purely in our heads, and owned by the individual. Practically, this approach is behind almost everything useful in our universe — like the spam filters processing this email. Engineers have long embraced the Bayesian model as it produces great results. Children learn in this way (see this TED talk) — constantly trying different hypotheses and updating their prior.
If by statistics, people mean mean quantitative measurement, I don’t have any issues with the use of quantitative analysis (like computing mean, variance, etc.). If by statistics, people include the Bayesian approach of being logical about uncertainty, I am thrilled with statistics. But if, by statistics, people include the frequentist approach of claiming statistical significance, using confidence intervals, etc., then I have several philosophical and practical concerns, as outlined in my paper.