Flawed Research by Andrew King
After I published one of my replications of an influential published study, a colleague asked me if I was being unfair: “How many publications would face up to such scrutiny?? If we set our standards so high, won’t we find that many of our famous studies are flawed?”
Yes. Exactly. ?I think we take most “famous” studies way too seriously.? Science is hard, and making progress toward the truth is even harder.? At best, most studies are flawed hints at the direction toward some truth, but they get interpreted as revealed fact.?
I know this most precisely by comparing what I think we know from my most famous study and what I hear people say about it.? I’ve heard in seminars statements like this: “We know from King & Lenox (2001) that industry self-regulation doesn’t work”.? BUT WE KNOW NOTHING OF THE KIND.? Mike Lenox and I did some statistical tests and rejected the NULL hypothesis for a few hypotheses aligned with the idea that one program wasn’t working in one industry for one outcome variable in one sample from a population taken at one point in time.?
Mike and I found, at best, a hint at a direction toward truth.
And we made mistakes.? Some of which we caught in time and some we did not.? Our main variable captured relative waste production, so positive coefficients meant worse performance.? This caused us to get our analysis wrong several times, so we wrote on the whiteboard “POSITIVE IS BAD” as a reminder.
We didn’t catch at least one serious and embarrassing error.? Since we wanted to compare members of the American Chemistry Council’s Responsible Care Program to non-members, we needed to account for the fact that the treatment (RC membership) was not randomly assigned.? We didn't do this well, and thus ended up with estimates that may confound the treatment and endogenous choice.
领英推荐
Our study has been replicated, with mixed results, by two brilliant scholars Shanti Gamper-Rabindran & Stephen Finger.? They seem to think we got it right concerning the effect of the program on pollution, but they think we got it wrong concerning its effect on accidents.? The latter is closer to the program’s intent, so it is probably more important.? Finger & Gamper-Rabindran (2012) find (a hint) that the ACC’s Responsible Care Program “reduces the likelihood of accidents by 2.99 accidents per 100 plants in a given year.”? That’s a big deal.
Let’s review: in my most famous study, my coauthor and I made a serious mistake, but may have gotten the right hint concerning one outcome, but probably got it wrong concerning a more important one.? That’s science.
What is not science is that because our paper has 2400 citations it is now known to be true and thus should never be replicated or found to contain flaws. ?
?
Auden Schendler Jitendra Aswani Michael Lenox Mike Barnett Mike Toffel Alex Edmans Tom Gosling Tom Lyon John Sterman Florian Heeb Florian Berg Lisa Sachs Bill Baue Laura Marie Edinger-Schons Prof. Dr. Ali Aslan Gümüsay Duncan Austin Luca Berchicci Ken Pucker Brent Goldfarb David Kirsch Nilanjana Dutt Gianmario Verona
Artist of Yr ’23-25; CSO Amb. @UNECOSOC; UNSDG Hub Kenya Exec. Dir; consult Nobel Peace Prizes 2022, '10, '07; Nobel Women 2012; #UN #I4T#T7 #G21; Hon. Adj. Professor, Scot Green Party ; #Lancmag writer.Views=my own
3 个月Andrew King Not sure why your post keeps appearing in my thread, probably because we have some contacts in common or linkedin is trying to sell me something using you...lol. Anyway I read your blog... Hope you get the replication results you need with humour :) ! However, I have to say, many of my research are NOT replicable...e.g. mixed methods on specific dynamics inside World chess championship, game of the century (longest game in recorded chess history), under global covid lockdown, during Dubai World Expo with Indian World chess champion AND Ukrainian Female World chess champion Grandmasters only in a specific way, in a specifc field context. So in some cases, the world will just have to live with the fact that people like me, or Jane Goodall or Newton or Einstein or whoever, did our best to be rigorous given the uniqueness of Real life in the field. I know this is probably too Anarchic Idealists or just Keynesian Economists about it, however practice and field research, just isn't a nice controlled lab and it never will be. We can't always please the funders. Sometimes it has to be love of Research and Science for its own sake! Hope we can agree on this.
Professor of Leadership & Organizational Behavior
5 个月this post reminds me of why I miss talking with you, Andrew King!
Chief Investment Officer, ClimeFi | ex-WEF & ex-Masdar | Carbon Removals | Blended Finance & Sustainable Finance | Climate & Development | Public-Private Partnership | SDGs | Angel Investor | MBA
7 个月So much respect for my former professor Andrew King being humble and forthright about the fallibility of his research. In this current era of absolutism, is is refreshing to see an acknowledgment that science may evolve and that an error or mistep should not invalidate all findings or signal a malevolent agenda.
Shelby Collum Davis Professor of Sustainability, Fletcher School of Law and Diplomacy, Tufts University
7 个月This is really excellent - scholarship can benefit from us taking a self-critical view on published work.
General Counsel Corporate Investments & Institutional Banking
7 个月Adore this as we do know statistics are what we make of them, but once we appreciate we have our bias tracking that can help us. Is it the jgnorance that gives us artifical bliss?