Setting standards for the Gold Standard: why we need to appraise systematic reviews

Systematic reviews as we know them today arose in response to (among other things) Archie Cochrane’s critique of the medical profession:

It is surely a great criticism of our profession that we have not organised a critical summary, by speciality or subspeciality, adapted periodically, of all relevant randomised controlled trials (Cochrane 1979).

The rationales for systematic reviews are many, perhaps the most persuasive are that they prevent us from cherry picking studies, and they enable us to appraise the quality of a relevant body of evidence. It is common sense that we should not base decisions about which treatments to use on cherry picked studies.

Cochrane was particularly critical of obstetricians and gynacologists, to whom he gave a ‘wooden spoon’ (which is the opposite of a gold medal, given to the losers of maths competitions at Oxford and Cambridge).

Taking up the challenge, Iain Chalmers, Murray Enkin and Marc Keirse organised the field of obstetrics and gynecology, and produced a book called Effective care in pregnancy and childbirth.

Cochrane gracefully removed the wooden spoon and noted in his preface to the book:

The systematic review of the randomized trials of obstetric practice that is presented in this book is a new achievement. It represents a real milestone in the history of randomized trials and in the evaluation of care, and I hope that it will be widely copied by other medical specialties.

Four years later, the international Cochrane Collaboration was established, following the opening of the first Cochrane Centre in Oxford, UK in 1992.

The Collaboration’s aim was to do more of what Chalmers, Enkin, and Keirse did by organising the evidence to answer specific questions in a systematic way by conducting ‘systematic reviews’. Systematic reviews quickly replaced expert consensus as the ‘gold standard’ way of deciding whether treatments worked.

Now, it would be all but unheard of for national guidelines and clinical decisions to not appeal to systematic reviews as justification for their decisions.

Recently, however, systematic reviews have become a victim of their own success. As I write this, there are over 30,000 systematic reviews published each year. Between 1986 and 2015, the number of systematic reviews has grown over 2500% (compared with all items indexed on PubMed). Professor John Ioannidis estimates that only 3% of these are (a) published, (b) of decent quality and (c) clinically useful.

(Picture from here)

The over-production of misleading reviews presents a problem. Systematic reviews inform guidelines and clinical decisions, yet only a small proportion appear to be trustworthy. This problem is exacerbated by the fact that reviews of systematic reviews are becoming more common. Just as it is important to know whether the individual studies that make up a review are reliable, it is now important to know whether the individual reviews in the reviews are reliable. We cannot assume that because something carries the label ‘systematic review’ that it is reliable.

We cannot assume that because something carries the label ‘systematic review’ that it is reliable.

The solution to the problems with potentially biased reviews is to appraise the extent to which systematic reviews are trustworthy. It is standard practise to assess the risk of bias of randomised trials (and other studies), and systems for rating the quality of systematic reviews have existed for almost two decades https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1810543/, with ROBIS (Risk of Bias in Systematic Reviews) having emerged as widely accepted.

The task of appraising the quality of systematic reviews is, of course, easier said than done. Appraising reviews using ROBIS requires expertise, and because there are so many reviews published every year, substantial resources are required to appraise even a substantial proportion of them. Some databases are starting to provide collections of appraised systematic reviews. They are helpful for patients, doctors, and policy makers who wish to know the extent to which the 'gold standard' has unadulterated (low risk of bias) evidence of effectiveness. I predict that these collections of appraised reviews will become more common in the near future.

Reference for which there is no hyperlink

Cochrane AL (1979). 1931-1971: a critical review, with particular reference to the medical profession. In: Medicines for the year 2000. London: Office of Health Economics, pp.1-11.