The Very Human Lesson in Psychology's Replication Crisis

We are turning more and more to science to enlighten us, unify us and save us. But what happens when science itself becomes fake news?

The superstars in the leadership and personal growth movements of the 21st century have been scientists — the psychologists who are researching human potential and human dynamics. Their findings have been popularized in the media, TED talks, books and wildly popular college and business school classes. In my own class at Columbia on Personal Leadership & Success, I’ve actively tapped this research.?

But today, psychology is facing a grave credibility problem, called the “replication crisis”. If you like taking in scientific advice on emotions, behavior, happiness, relationships, influence, leadership and more, here are a few things you need to know about this crisis and how to protect yourself from it.?

A psychologist will typically run an experiment that yields a certain result, and then publish this research in a scientific journal. For instance, one psychology study has shown that if you do a “power pose” before an interview or other high-stakes meeting, you’ll up your confidence level. Another study has shown that there are typical facial expressions we all make when we experience a certain emotion like anger or fear. A third study — the classic Milgram experiments at Yale — has shown that, under a bit of pressure from an authority figure, most people can be made to perform pain-inflicting actions on others.?

Once these studies are published, the media catches on, the author(s) are invited to give talks and write books, and the idea catches fire in social circles and corporate L&D groups.

Until recently, no one had questioned the legitimacy of these studies. Then in 2015, ?Aarts?et al. decided to launch a project where some of the most prominent psychological experiments would be replicated by other researchers. This is after all the core of the scientific method — that an experiment, well designed and executed, should yield results that can be replicated anytime in the future by other experimenters. Otherwise, its findings wouldn’t be credible. After replicating over 100 of these studies, these researchers have reached a startling conclusion.

More than half — half! — of these studies are not replicable. In other words, we cannot validate them.

Why, you might ask, are these studies showing different results? There could be at least three reasons.

In some cases, it may be that the research was done under certain conditions — based on the subjects used, the instructions that were given, or the context in which the subjects were placed — that influenced the research results, and that could not be replicated in the replication studies. This is the most innocuous reason, and, one would hope, the most common.

?In other cases, as some scientists are acknowledging, this is due to more questionable tactics used by researchers. You come up with a hypothesis and you do an experiment to prove it. If the experiment does not yield the desired result, you scrap it, and do another one. And you keep repeating this until your experiment gives you the desired result. You then report only this data from the successful experience in your research paper that you submit for publication — the past failures are not included. Anyone with a basic knowledge of statistics will tell you that if you repeat an experiment a large enough number of times, at some point it will give you — randomly — the result you are looking for. They call this publication bias.Another troubling practice that some claim is used at times is p-hacking — a way to find seeming patterns in data where none might actually exist.

Perhaps the most troubling is the third reason. This is when there is evidence that the psychologist has, consciously or unconsciously, directly influenced the subjects involved in the experiment, and/or manipulated the data, in order to get the desired outcomes. Recent investigations on two of the most famous psychological experiments — the Stanford Prison Experiment by Zimbardo and the Yale Obedience Experiment by Milgram — have revealed serious issues with how the lead scientists manipulated the experiments and their findings.?

?So if half of all the science is not replicable, does this mean you will have to dispense with half the scientific knowledge you’ve been using to guide yourself and coach your leaders and your teams? Unfortunately, the course-correction is likely to be much worse than that.

Here’s why.

A recent study has shown that those psychological experiments that have been shown to be unreplicable are more than 150 times more likely to be cited in research articles than the experiments that have been replicated. The psychological community, in other words, is drawn to novel results — which are not replicated — rather than broadly tested and validated results. Given how much more these non-replicable studies are cited, they are of course much more likely to be read and popularized and covered in the media — and thus, more likely to make it into popular consciousness.?

?And here’s the final nail in the coffin — even after results are published about a study’s non-replicability, only 12 percent of papers that cite that study are likely to acknowledge this non-replicability. The other 88% percent of times this study is cited, there’s no reference to the fact that this study was found to be non-replicable.?

So fake news in psychology exists, it’s more than 50% of the studies, it’s most of what is cited and popularized, and even after it’s shown to be non-replicable, it’s still actively cited and used.?

It is a humbling moment for science and academia. But I also feel it is a very human moment. Because it shows that science is ultimately a work-in-progress, a frail human enterprise infused both with noble aspirations and self-serving motives, dispassionate analysis and biased reasoning, selfless strivings and ego-driven manipulations.

While we have actively used psychological research findings in our leadership training at Mentora Institute and in my Personal Leadership class at Columbia, I recognized early on that not all the science was compelling, mature, or credible. I triangulated it with other ways of knowing — studies of inspiring leaders from history, timeless spiritual wisdom across cultures, and clients’ and participants’ lived experiences. I sought to connect the dots with other scientific disciplines like neuroscience and psychotherapy. That way, I am glad to say, much of what we’ve built in our leadership curriculum continues to hold true even as psychological findings are being challenged and, in part, invalidated.?

Have you been drawn in recent times to ground your personal development — and your organization’s leadership development — in science, and does this replication crisis leave you a bit shaken? If so, here are some suggestions I can offer, based on the path my team and I have taken at Mentora | Columbia:

1. Don’t have blind faith in “science”. There’s good science and bad science. The scientific method is deserving of our respect, but the scientific enterprise is a human enterprise, prone to bias, ego and attachment. And therefore…
2. ...distinguish between scientific findings that are based on a large number of well-replicated studies — like the findings of cognitive behavior therapy — and those that are based on one or two highly publicized studies that have not been broadly replicated.?
3. When you learn something in the media or from an expert about a psychological experiment that shows XYZ, ask yourself, “How does XYZ align with common sense? With my lived experience? With the spiritual wisdom of the ages? With what qualities and behaviors I see in the people who inspire me?” Only trust the result if it resonates with you at your core.
4. Before you accept something as truth — think this way, act this way, feel this way! — try it out for yourself and carefully assess the impact. If it doesn’t work the first time, then learn, improve, and try it again. But don’t keep believing it if you aren’t ultimately getting results, and don’t teach it to others without validating it for yourself.?
5. Look for corroborating evidence from other disciplines that involve a different class of experts - neuroscience, medicine, psychotherapy, historical studies of inspiring leaders, and more. In our training programs for clients at Mentora, we embraced Carol Dweck’s Growth Mindset, for example, because it was based on a wide range of studies, because it was very aligned with the spiritual truths of all great faiths, and also because we saw it reconfirmed by the science of neuroplasticity.

Here is my final thought. We have, justifiably, elevated science on to a pedestal, respecting its objective, fact-based, logical seeking of truth. We now need to realize that behind the enterprise of science lies the judgements, actions and motives of human beings. When the human beings are flawed - driven by a need for self-promotion, unconsciously biased in their thinking, attached to a particular result - the science, too, will be flawed. In other words, we don't get to produce great scientists solely by having people excel at math, science, psychology and other academic disciplines.

Great scientists have to be, at their core, great human beings, driven by a purity of motive and a receptivity to truth in whichever form it arrives at their door. And, I would offer, that's true for all other professions beyond science as well. It is thus true for you and for me in our own work too.

?