Hypothesis test, significance test, and confidence intervals

While working on our textbook about 15 years ago I looked into the history of concepts like confidence interval, hypothesis and significance test, and had many discussions with Leo Held about the topic. My impression is that in the current debate folks are voicing their view without reference to the history and proper understanding where concepts actually come from. To quote Brad Efron:

Those Who Ignore Statistics Are Condemned to Reinvent it.

So, let’s get started:

Fisher "invented" p-values to quantify the evidence against a null hypothesis of no effect. For him, they were a tool independent of a given test and sample size which allowed combination (meta-analysis anyone?) of evidence from multiple different experiments to finally arrive at a final conclusion about a scientific question based on the totality of evidence. He called this concept SIGNIFICANCE test.

Neyman and Pearson proposed a framework for making a decision between a precisely specified null and alternative hypothesis. Whenever you make an explicit decision you run the risk of that decision being wrong, and N-P wanted to quantify that risk. This framework is referred to as HYPOTHESIS test. If you reject the null you call this "statistically significant". If you remember how you learned to make the decision in a hypothesis test – comparing a test statistic to the quantile of the distribution of it under the null – no p-value is needed at all.

Finally, with a confidence interval we assess what true populations values are compatible with the data that we have observed.

It should become clear that these three concepts were initially meant to address very distinct goals. So, where does the confusion come from?

• You can make a decision in a hypothesis test using either a p-value (the infamous p <= 0.05) or using the confidence interval (is the null outside of the CI?). However, that was not at all their initial intention.
• Of course, it does not help that we call the result of a HYPOTHESIS test “statistically significant” when Fisher called his concept of p-value SIGNIFICANCE test.
• A lot of what we call "science" is incentivized to make claims, and it has become easy to separate "findings" from "non findings" by simply comparing some p-value to the arbitrary threshold of 0.05, although that mixes the concepts of hypothesis and significance test and completely ignores the issue of multiplicity induced through the former.
• p-values are often used to coin "findings" as "statistically significant". This mixes the concepts of significance and hypothesis test and this has never been in the sense of each concept's "inventor". I therefore consider it odd, to say the least, to "ditch" p-values because they are not used properly. Rather, (1) statisticians need to use concepts properly and (2) educate science as a whole how these concepts should be used and what their respective properties are. I know, not easy, but still.

So, when to use what? In my opinion, a hypothesis test only makes sense (if at all) in the context of pharmaceutical drug development. There, you only start a Phase 3 trial based on sufficient prior synthesized evidence and you very carefully select the primary endpoint. Furthermore, relevant stakeholders, primarily regulators, have a genuine interest in controlling false-decision probabilities, notable the type I error. Hypothesis tests have proven useful in providing a pre-specifiable framework that leads to reasonably sized trials.

However, in science at large, this is not how we generate evidence. Rather, to quote Sterne and Smith (2001)

In many cases published medical literature requires no firm decision: it contributes incrementally to an existing body of knowledge.

Furthermore, Blume and Peipert (2003):

The reporting of scientific results is not about making decisions, but about collecting, summarizing, and reevaluating evidence.

So what do I recommend?

• Unless we are really talking about a pre-specified hypothesis test with interest in one single pre-specified hypothesis we should not use the term "statistically significant". Specifically, when doing some exploratory analysis and looking at many p-values we should not assign this label to those below a certain threshold. Potential qualification of evidence based on p-values, see above picture, is given in the paper by Pocock et al (2015).
• Use "statistically significant" for a clearly defined hypothesis test that actually entails a decision, like in pharmaceutical drug development (acknowledging that a significant hypothesis test for a primary endpoint does not mean "approval", but is at least an entry ticket for negotiation with regulators).
• The decision about "significance" in a hypothesis test is a binary one. This means a hypothesis test is either significant or not. Labels like "highly significant" or even much worse something like "trend to significance" are bogus and should not be used. Or is a p-value of 0.04 a "trend towards non-significance"?

That’s a wrap! I leave the discussion of “relevance” vs. “significance” and how Bayes plays into all that for another day.