Figure 2 provides the first scientific proof that clinical decision support improves medical imaging appropriateness scores in the United States

Thank you very much for all of the emails and calls about the manuscript. Here is my best explanation for Figure 2 in our manuscript that can be found here https://rdcu.be/cfQI8

Figure 2 has important distinctions from Figure 1. Before I start, please remember the Take Home Messages that I have uploaded so far:

Frankie Take Home Message #1: 57% of advanced imaging requisitions are appropriate!  Great news and room for opportunity. 

Frankie Take Home Message #2: Almost 15% of advanced imaging requisitions are gray – that means they did not map to an appropriate use criteria (AUC). Way better than the Demonstration Project.

Frankie Take Home Message #3: The X-axis is not time and future CDS / EMR research that implements this new methodology is very promising.

Take Home Message #4: CDS does work and the benefit of using it and sticking with it is better appropriateness scores!

The first thing that is worth nothing is that Figure 2 had 0.95 confidence intervals and shows (very high) statistical significance. Like in Figure 1, the X-axis is the ordering physicians experience (what happens over 200 reps) using clinical decision support. It is not time. If you read the first post (description of Figure 1), you might remember that I noted there was no statistics for Figure 1. 

Now I will explain why... 

Please if you would like, look at Figure 1D. Here is a shortened link that Journal of Digital Imaging sent to me, if you need it: https://rdcu.be/cfQI8

In Figure 1D, you see that for all 7M+ advanced imaging requisitions, there are many more providers on the left-hand side of the curve – people have mistakenly suggested that this means people choose not to use CDSM. That is not the case in our medical imaging CDS dataset, and we can address this with more data in future papers. What Figure 1D does mean is that many people (medical students and other people who rotate though services, nurse practitioners, many specialists who do not get lots of imaging) order a small number of studies. Stay tuned for more details on this, but suffice it to say for now that the reason that we don’t – and can’t – do statistics on Fig 1 is that the composition of each bin on the X-axis is different in Figures 1A, 1B, and 1C.

Here is a marked up version of Figure 2. Note that the green, yellow and red rates are color coded on the right hand side.

Figure 2 solves that issue. In Figure 2, we still look at the whole data set, but we ONLY include those providers who have completed at least 200 “reps” with the CDSM. Examining an unchanging set of 10,123 providers with ≥ 200 advanced imaging requisitions (2,024,600 total requisitions) allows us to compare bin 1 (the first 10 reps) versus bin 20 (the last 10 reps) and compute statistical significance within a fixed set of providers.

What the data shows is that the significance is (overwhelmingly) strong and we can say with confidence that the green rate goes up by about 3% while the red rate goes down by about 3%, for a total net change of 6%. For Figure 2, the yellow rate does go down a bit – less than 1% but still significant.

To carry on with the analogy of the gym – Figure 1 looks at everyone who went to the gym after a “New Year’s resolution”, and we just looked at them during their last session and we saw how they did. Figure 2 still looks at everyone, but it only studies with statistical rigor those people who went to the gym 200 times or more. That gives the equipment (CareSelect from Change, using all AUC from the American College of Radiology over all instances for 3 years) a fair (ie statistically valid) assessment. If we simply assumed that getting a gym membership itself had somehow improved physical strength, we would have been incorrect – and yet, this is unfortunately has been done in multiple studies assessing CDSM up to this point.

So here is next point (Take Home Message):

Frankie Take Home Message #5: CDS really does improve scores – when you follow the EMR experience (again, do not think of time since implementation) of the people who ordered CT, MRI, US, and nuclear medicine studies, there is unquestionably more appropriate scores for studies being performed in the United States as a result of CDS use.