What are common errors to avoid when assessing statistical significance?

1 Misunderstanding p-values

A p-value is the probability of obtaining an effect at least as extreme as the one in your sample data, assuming the null hypothesis is true. A common error is misinterpreting this as the probability that the null hypothesis is true or the chance of making a mistake by rejecting the null hypothesis. Remember, a low p-value indicates that your results are unlikely under the null hypothesis, not that the null hypothesis is definitely false or that your alternative hypothesis is true.

2 Ignoring effect size

While p-values can tell you if an effect exists, they don't convey the size of the effect. This is where effect size comes in, providing a quantitative measure of the strength of the phenomenon. A statistically significant result with a minuscule effect size may not be practically significant. Therefore, it's important to consider both statistical significance and effect size to understand the real-world implications of your findings.

3 Overlooking assumptions

Statistical tests are based on certain assumptions about the data, such as normality, homoscedasticity, or independence of observations. Ignoring these assumptions can lead to incorrect conclusions. Before you perform a statistical test, check that your data meets these assumptions. If they don't, consider using alternative tests or transformations that are more appropriate for your data.

4 Multiple comparisons issue

Conducting multiple statistical tests increases the chance of finding at least one statistically significant result purely by chance. This is known as the multiple comparisons problem. To avoid this, you can adjust your significance level using methods like the Bonferroni correction or control the false discovery rate. These adjustments help to maintain the overall error rate and give a more accurate picture of statistical significance.

5 Sample size neglect

The size of your sample can greatly influence statistical significance. A very large sample might lead to statistically significant results even for trivial effects, while a small sample might not have enough power to detect a real effect. It's essential to calculate and use an appropriate sample size: too small and you risk missing the effect; too large and you might waste resources or detect inconsequential differences.

6 Data dredging pitfalls

Data dredging, also known as p-hacking, involves searching through data for patterns or testing multiple hypotheses without a pre-specified plan. This increases the likelihood of finding significant results by chance. Avoid this by setting out your hypotheses and analysis plan before examining the data, and resist the temptation to keep testing until you find something significant.

7 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?