Scholarly Publishing - Peer Review's Big Problem

The Big Problem with Peer Review in Scholarly Publishing

Adam Mastroianni and Russ Roberts do a really nice job of covering the big problem that peer reviewed journals have in the latest episode of EconTalk, namely that peer review just doesn't do what we expect and intend. The public's commonly-held view of peer review is that it is a reliable means of validation and trust, and that when a journal publishes a paper, it stands behind the validity of the work. It largely does not deserve that reputation. In practice, peer review is often a way to disqualify papers that don't meet the style, format, or English writing capability that is expected. This desk rejection usually happens in the first 15 seconds of glancing at a paper, especially at the title and abstract. In the main, journals use peer review to vet for how novel or interesting the work is rather than whether the work is correct (with the exception of the more recently invented "sound science journals"), and almost in no circumstance is peer review capable of reproducing the work sufficiently to stand behind the claims made in the paper. It is no wonder then that most research is not reproducible.

Peer review is only as good as the reviewers. With millions of new research papers submitted each year, and such a lack of reviewer time, we increasingly suffer from the problem that it is difficult to find good reviewers for a paper, and therefore the two referees are often from the 5th or even 10th batch of reviewers who were approached to give feedback on the paper. When a journal only has one or two referee reports to rely on, and you aren't getting anywhere near the best reviewers for a paper, our odds of the sorting algorithm of peer review filtering the right papers in and the wrong papers out is low.

In our work on Rubriq in 2016, Keith Collier and I identified that 15M hours per year are wasted in duplicative peer review, which Adam and Russ mentioned. This is only the duplicated reviews for papers rejected and submitted elsewhere. If you pull out the entirety of peer review as a formal step before publication, the time (and therefore cost) is much higher.

Publishing in Preprints is Free

In an era of internet, "publishing" a paper is just about getting the brand of a journal on your work. It is no longer about distribution, which was the core value proposition of publishing for the vast majority of time we have had academic journals. Even primacy of a research claim has now shifted to the free preprints like Research Square, now the largest interdisciplinary preprint in the world, which was led by Damian Pattinson , Michele Avissar-Whiting , Juliet Kaplan , Rachel Burley , and others.

Here is a brief history of scholarly publishing for reference. Adam and Russ remind us that we only started doing peer review since the 1970s. It is a shame, because the world wide web was invented by Tim Berners-Lee to enable information sharing between researchers. Sadly and ironically, this is the last bastion of publishing that hasn't reinvented itself in the internet era. Today we essentially have a PDF version of the print journal, in the same format and style, and not much more.

The biggest element holding back progress is the use of Journal Impact Factor in assessing the quality and contribution of the work, especially in tenure and grant decisions.

Where Do We Go from Here?

We are reaching breaking points in maintaining both the practice and illusion of peer review as a sign of reliability and validity. It is becoming apparent to more of the stakeholders in research, funding, and policy that the emperor has no clothes. When funding and tenure decision frameworks start to pivot away, the whole game opens up. We will need to find new ways of sharing, validating, and building on top of one another's work. Preprints are a necessary step along that path. I believe in the future model we will acknowledge that peer review as it stands today doesn't work as intended, that platforms can do more of the sorting and filtering work for us while cutting down the time and cost of publishing, that both Open Access fees and Library subscriptions are low-yield capital that can be better allocated, and that much more can be done to add value to the process in our internet era to explain and engage society in the work.

Next time I'll be sharing what I see as the three biggest opportunities in scholarly publishing.

Shashi

(all art was generated on DALL·E 2)

My substack: https://shashim.substack.com/

#peerreview #scholarlycommunication #researchjournals #researchpromotion #researchplatforms #scholarlykitchen #econtalk #rubriq #researchsquare #preprints