What are the common mistakes in interpreting p-values and how can you avoid them?

Mistake: Treating the p-value threshold (commonly 0.05) as a strict cutoff without considering the context or the study design. How to Avoid: Understand that the 0.05 threshold is arbitrary. Consider the broader context, including the study's design, sample size, and the potential consequences of type I and type II errors. Adjust the threshold if necessary, especially in exploratory studies or when dealing with multiple comparisons.

p-value is the probability of observing this data given that the null hypothesis is true. For example, if the null hypothesis is no difference between the groups, what would be the probability of observing this data? We can look here at fisher's disjunction: either a rare event happened or the null hypothesis is false. Therefore, there is no magical boundary, only the understanding of how rare the event was given the null hypothesis is true. The threshold of 0.05, was proposed by Ronald Fisher so he would see rare event given that the null hypothesis is true in 1 out of 20 experiments. This was a convention rather than a strict boundary.

Mistake: Misinterpreting the p-value as the probability that the null hypothesis is true. How to Avoid: Recognize that the p-value indicates the probability of observing the data, or something more extreme, assuming the null hypothesis is true. It does not measure the probability that the null hypothesis is true.

A common mistake is to treat p-value as probability that the null hypothesis is true given the data, that is P(H0|Data), which leads to wrong interpretations. Actually its the opposite, its P(Data|H0). p-value is closely related to Type I Error (but is not directly the Type I error rate), by showing how rare the is the data observed under the null hypothesis. Therefore, p-value can be seen as the rareness of the data observed, which help us to accept of reject the null hypothesis.

There is not mathemagical formula for every test under every condition. Your statistical test is always suited for a specific case. You need to check whether it is the appropriate tool for the work. You can check some assumptions through other statistical tests; for example, you can check if your data is normally distributed by using a komolgorov-smirnov test. It is more worthwhile to check these assumptions with statistical tests than to use volatile plots sometimes. Usually statistical packages already run these tests for the assumptions, and if the assumptions are being met, it can be seen directly in the test output.

Mistake: Ignoring the assumptions underlying the statistical test used to calculate the p-value (e.g., normality, homogeneity of variance, independence). How to Avoid: Always check and validate the assumptions of the statistical tests you are using. Use diagnostic plots and tests to verify assumptions. If assumptions are violated, consider using alternative statistical methods that do not rely on those assumptions.

The probability of, for example, the uplift observed, is different than the actual uplift. We can have a low uplift with a low p-value, and high uplift with a high p-value and so on. The p-value must be taken as rareness of the result if the null hypothesis is true, and not the uplift itself.

A common misinterpretation is to view the p-value as the probability of making a mistake by rejecting a true null hypothesis (a Type I error). This is incorrect

Mistake: Focusing only on studies with significant results, leading to a publication bias. How to Avoid: Value and report non-significant results to contribute to a more balanced scientific literature. Consider pre-registering your studies and hypotheses to enhance transparency and reduce bias.