A Manager’s Guide to Making Machine Learning Work in the Real World

There’s plenty of coverage on what machine learning may do for healthcare and when. Painfully little has been written for non-technical healthcare leaders whose job it is to successfully execute in the real world with real returns. It’s time to address that gap for two reasons.

First, if you are responsible for improving care, operations, and / or the bottom line in a value-based environment, you will soon be forced to make decisions related to machine learning. Second, the way this stuff actually works is incredibly inconsistent with the way its being sold and the way we’re used to using data / information technology in healthcare.

I’ve been fortunate to have spent the past dozen years designing machine learning-powered solutions for healthcare across hundreds of academic medical centers, international public health projects, and health plans as a researcher, consultant, director, and CEO. Here’s a list of what I wish I had known years ago.

The machine learning is a capability, not a solution. Machine learning is math that we learned how to automated (i.e., software) that allows us to analyze, optimize, customize, and prophesize in new and powerful ways. We can use machine learning to discover what needs to change and how best to change it.

A solution is a very different thing. It requires getting folks to do something differently, tracking those differences, relating them to outcomes (good or bad), and sharing that information back with the team. For managers, machine learning minus change is at best an innovation project and at worst a waste of resources. Machine learning + change can be a powerful solution.

Improvement is not installed. I recently met with a data scientist at a large healthcare org that created a fantastic model (“great performance, highly predictive”), handed it off to "the business," and lamented the fact that it didn't make a difference. He quickly followed with, "It's dangerous to assume that the success of a data scientist is dependent on whether a model is used." It’s dangerous to pretend that it isn't.

If you think machine learning is a thing for IT to install or hand off, you’re doing it wrong. Amazon doesn’t install a third party population book selling algorithm. Target doesn’t use the same software as all of their competitors to learn how best to manage their logistics. The opportunity is to move beyond one size fits all. It’s about learning from data and improving. Learning and improving are processes, not products.  

Pick a $5M problem. Change is hard. Pick a problem that will garner enough attention and resources to achieve change. Our rule of thumb is we want projects that lead to $5M in new revenue or cost savings but 5 is just a heuristic. Whatever number is large enough to get & keep the CEO’s attention is the right number.

There’s another implication here. Spend the time doing the math to know how much potential value there actually is. It’s a lot cheaper to discover early that there isn’t enough value than to get months into a project and learn that your project isn’t all that important. Trust me on this one:)

It’s an executive (C suite) decision to proceed. At Cyft we have learned to require not only exec sign off but bi-monthly check in meetings to keep things on track. If you picked the $5M project, helping you succeed is worth 30 minutes of their time every other week so they can clear barriers.

And a team sport to execute. Successful execution requires a committed multidisciplinary team. Our must have list includes representatives from IT, business analytics (someone that knows the data), the business (or clinical) owner, and a project manager. This working group meets weekly and is run by a project plan with milestones and timelines. This isn’t rocket science but we have found that this level of discipline is necessary for keeping a project on track.

If you build it, they’re unlikely to come. However, if you built it with them based on their needs, calculating potential ROI, informing them of the trade offs, and involving them in decision making, they’re likely to support your combined efforts to address some of their most important challenges.

No one cares about your c-stat. I have yet to meet an executive willing to measure returns in terms of p-value, c-stat, F-measure, or any of the statistics that researchers are judged by. If you’re using machine learning in an enterprise environment you will be held accountable to a return on investment.

The good news is that a solid understanding of the business + basic arithmetic + model performance allows one to bridge from interesting statistics to real ROI calculations. The bad news is it’s yet another important task that must be planned for that has little to do with machine learning itself.

Don’t underestimate the importance of education. This stuff works differently than people are used to. Clinicians that have spent decades becoming masters of triage will be asked to focus on people whose need isn't overtly obvious. Managers that are used to telling their IT teams exactly what reports they need will now be shown the data they should care about. And all because a computer said so.

If you don't invest in helping people understand how this works and foster trust in your approach do not expect them to simply adopt your results.

Clear, concise storytelling is critical. I have been at this for 15 years and I'm still constantly searching for metaphors, analogies, common ground that will allow me to bring people up to speed on how this works, the results we got, baseline versus new ROI, etc.

Integrate or bust. Every healthcare organization has a nearly unique combination of information systems and workflows. Absolutely none of them wants a new interface / system to log into or a new parallel workflow to keep track of. Find a way to integrate.

Keep learning & improving. Improvement is not a binary thing that did or did not occur. Yet so many data-related projects seem to assume as much in their execution. If ROI is the goal then agree to baseline, activity, and outcomes metrics as soon as the problem is defined. Measure them often. Change practice accordingly. Otherwise, it's just machine learning - not human learning.

The good news is, this stuff is as doable as it is important. With a clear focus on the problem to be solved (hint: it’s not ‘use machine learning’), a dedicated team, and disciplined project management, your team will begin to capitalize on the tremendous gains experienced by nearly every other industry. 

Dr. Leonard D’Avolio is an Assistant Professor at Harvard Medical School and Brigham and Women’s Hospital and the founder and CEO of Cyft, a company that helps healthcare organizations become learning healthcare systems. As part of his role, he writes, speaks, and consults on the topic of making data work for healthcare and he shares lessons learned on Twitter at @ldavolio and on Cyft’s website (cyft.com).

#education #storytelling #graphics #statistics #machinelearning