30 Helens Agree: Reproducibility Matters

… and now for something completely different! 30 Helens. Back in the day, there used to be a genius Canadian comedy show called “The Kids in the Hall.” Among the memorable recurring skits (including the chimeral Chicken Lady) was one of the bookends to the show, called “30 Helens agree”, where 30 Helens discussed deep thoughts like love hurts, honesty, cole slaw, a good pair of shoes and tattoos and other pithy maxims.

If The Kids were producing a show today, perhaps human chimera for organ harvesting with the Chicken Lady would be on the roster along with my take on circular references with the 30 Helens agreeing that “Reproducibility Matters.” Getting 30 Helens together first, then having them agreeing on anything is a pretty low probability, especially in the US – from the early 1970s the name “Helen” has not been on the top 200 most popular names. The name “Helen” was 0.039 percent of total female births in 2015 according to the US Social Service Administration. As an estimation exercise, you can determine the average age of each Helen in the picture.

Reproducibility is more than just agreeing on urban memes. Wikipedia defines "reproducibility" as "the ability of an entire experiment or study to be duplicated, either by the same researcher or by someone else working independently." I would specifically include data reproducibility within the definition. The dogma of science is best summarized by the eminent sociologist Robert K. Merton, who coined the phrase “unintended consequences”:

“The ethos of science involves the functionally necessary demand that theories or generalizations be evaluated in [terms of] their logical consistency and consonance with facts” Robert K Merton, Philosophy of Science Vol 5, nr 3, p. 326, 1938

The Global Standards Institute estimated (in 2015) that about half of all cumulative pre-clinical research in the US is irreproducible and costs about $28 Billion overall. The four categories of irreproducible research are charted below: 

I have noted many times that the National Institutes of Health (NIH) budget between 2012 and 2015 was about $120B, whereas the combined budget for Health and Human Services (HHS), Centers for Disease Control (CDC) and Food and Drug Administration (FDA) was about $61B – about half the budget for the translational areas! Is it this inequality that almost tripled the number of retractions of studies from major journals and has spurned sites like the Retraction Watch?

Two of my heroes at the intersection of biology and reality happen to be at Stanford University: Dr. John Ioannidis and Henry (Hank) Greeley. They are both the down-to-earth and humble intellectual and philosophical powerhouses in our world of complexity in biomedical research. Dr. Ioannidis very recently co-penned “A manifesto for reproducible science” with the threats to reproducibility in a circular flow shown below (and highlighted in red):

HARKing stands for "hypothesizing after the results are known.” Another effort of his is the Reproducibility Project for Cancer Biology. Hank Greely is one of the foremost thinkers on Healthcare and the privacy ethical, legal and societal implications of science upon it. His recent paper on the collision of science and privacy is a great read. Reproducibility has large societal implications that must be considered.

The Design of Experiment (DOE) guidelines describe Method Planning, Risk Management and Quality. We need to incorporate reproducibility techniques into DOE. Phil Green created the Q score for Genomics for each base call of the genome. We need a similar “per step” quality score for each experimental method in the process. The Nature journal has devoted an entire special (and recurring) issue toward reproducibility. I meet with exceptional scientists and clinical researchers on a regular basis – we need to inspire this and the next generation of curious minds that there is bias involved in science – asking the right questions and designing good studies are sometimes more important than the science itself. This is succinctly described in a recent Nature Behavior editorial below:

Building a framework to support this initiative requires a critical mass that includes physicians, patients and common citizens -- not just researchers and scientists. It will also require a public archive (for structured and unstructured data) and technologies to vet, question and reproduce both small and large scale studies. I would suggest a distributed, online Robert’s Rules of Order style citizenship governance with an AI organizer that includes DOE which connects to a large, available archive for verification and reproducibility studies for the two stages and microservices for each of the boxes noted in the figure above. It is worth noting that negative results from pre-clinical studies and clinical trials are as important as reproducibility. The PLoS Missing Pieces initiative is a great example.

30 Helens agree: reproducibility matters. Let’s hope all the Helens make reproducibility as beautiful as the Helen of Troy…

##