Acknowledging Research Limitations

In the midst of a replication crisis, I think one of the biggest problems facing researchers is not acknowledging the limitations of the research.

No research project is perfect. They all have boundaries and limitations that mean they might not be applicable in all situations (among other potential limitations).

I used to tell my research methods students I'd never seen a perfect research project, and if they did a perfect study, I'd sabotage it to keep the streak alive.

I have yet to sabotage a project to keep the streak alive.

Why wouldn't researchers want to acknowledge limitations? For one, they might think it would lessen people's perceptions of research, but if we acknowledge the aforementioned postulation of no such thing as a perfect research project, then we should have less fear of this kind of perception.

A scarier possibility is that the researchers might not know the limitations of the research, but that's a very different problem.

Acknowledging limitations of research is inherently hemming and hawing instead of moving forward with actions, but the alternative of moving forward with limited information is more dangerous, and we're paying for that fallout now.

The replication crisis is an issue of trust. Overstating findings is a further breach of that trust. Both I think can be prevented by acknowledging limitations.

For the replication crisis, limitations tell us we did a specific thing in a specific setting, and we need to know 1) if it'll happen again in similar situations, and 2) if it'll happen (or how it will change) as the treatment and the circumstances change. Nothing is universally good. We have to research things in different ways so fully understand how these constructs manifest in the real world.

For overstating findings, it's difficult to do so if you acknowledge the limitations. You found what you found, but let people know what it can't apply to. That's ok.

Again, there is no perfect research project. If you're ever in danger of doing the perfect research project, expect a visit from me. Until then, just acknowledge the limitations.