How can you identify statistical fallacies and biases in your field?

The way Statistics works is that there are mathematical and statistical fundamentals and then all the methods diverge from those fundamentals. To identify potential biases, one should always revert to the fundamentals and be aware if those fundamentals are included in the correct way. Learn about the axiomatic thinking, understand probability theory in detail and how to actually understand the data itself.

Know the mathematical fundamentals. Whatever it’s called, the science of data analysis is rooted in the same mathematics. Tools cannot substitute thinking.

Cultivate a strong preference to know what is real over a position. Statistician is not an advocate role like a lawyer. Lawyers zealously defend their clients’ interest. This means they take a position first and zealously defend it. Statisticians aim to infer generality and quantify uncertainty from data.

If something doesn’t make intuitive sense then it’s more likely (but not surely) to be wrong. It is also critical to investigate the data generating process i.e. how was the data created, in what context, who is being represented and what are their perceptions during their observation.

Understanding Statistical fallacies and avoiding them is most often related to not just the Statistical knowledge, but also the domain based knowledge. Understanding potential confounders, validation of the data, validation of the results, understanding limitations, understanding the potential latent variables are all vital in understanding potential statistical fallacies too.

The principle of "Extraordinary claims require extraordinary evidence" is always worth bearing in mind. It's more than just an invitation to scepticism: it is underpinned by Bayes's Theorem, one of the most useful tools in reasoning with data.

Due to the nature of statistics in combination with the law of large numbers, the more hypotheses you test, the more likely you are to find a statistically significant result even when there is none. This is the multiple hypothesis problem and several solutions exist like the bonferroni correction.

Perhaps the most common fallacy is mis-interpretation of p-values, and thinking that a low p-value means the chances of a result being a mere fluke are similarly low (their actual meaning is both subtle and often of no real interest). Also, never lose sight of the fact that p-values and Confidence Intervals only take into account random measurement error - not systematic error. There are many examples of amazingly "statistically significant" findings (p << 0.05) which turn out to be the result of some instrumental bias or other blunder (see eg the "Faster than light neutrinos" debacle of 2012).

An over reliance on achieving a statistically significant finding without considering the effect size is an all-too common mistake. I’ve seen published studies with a p-value of less than 0.001 but minuscule effect size(0.03). This is misleading in terms of the study’s key takeaway. It’s very important, if you’re going to use p-value that effect size is reported as well.

Be wary of conclusions based on huge databases. These can be riddled with selection biases that would normally be dealt with via the use of randomisation in collecting the data. "Big Data" sets often give impressively precise findings (ie v narrow confidence intervals), but which can be very inaccurate (ie lie far from the truth) because of very subtle biases lurking in the database. See eg Bradley et al Nature 2021 for a striking example of this, relating to Covid vaccination coverage.

The other aspect I find crucial in valid research is Identifying and addressing statistical fallacies and biases related to missing data assumptions. Key considerations include recognizing missing data, understanding the mechanisms behind it (Missing Completely At Random, Missing At Random, Missing Not At Random), and being aware of biases such as selection bias and non-response bias. Handling missing data can involve techniques like complete case analysis or imputation methods such as mean imputation or multiple imputation using chained equations (MICE). Transparency in reporting the missing data patterns, assumptions, and handling methods is essential for evaluating the impact on research outcomes.