How to make your AI applications ready for prime time
How deeply do you think about the real-world environments in which you will be deploying your AI applications?
A few of us at Stanford did a study on Google's AI ambitions in healthcare and the detection of "diabetic retinopathy" emerged as a successful case study. But now it seems that success was limited to the lab and that Google has had challenges in making it work in the real world.
In countries like India and Thailand, the number of diabetes cases is significantly more than there are doctors to test and detect early signs of diabetic retinopathy, which if left unchecked can lead to blindness. The tool that Google built can be a boon for such countries in effectively managing this condition. Only if it had worked as well in the clinic, as it did in the labs.
1. Adapt workflows for real-world environments:
Lab settings are controlled and often ideal, the world isn't. Solve for the lowest common denominator. In this case, the internet speeds in Thailand were significantly slower compared to the west and thus resulted in greater processing times and poor patient experience.
2. Test accuracy against a range of quality-of-data:
As most image recognition systems do, the deep-learning model here was trained on high-quality scans. To ensure high levels of accuracy, it rejected images that fell below a certain threshold of quality.
In Thailand, the nurses scanned dozens of patients an hour and often took the photos in poor lighting conditions. As a result over 20% of the images were rejected, creating additional work for nurses and inconveniencing patients to redo the scans.
3. Allow for Human Augmentation
This one is tricky. AI's core promise is automating human decision and reducing the errors in judgment and minimizing personal biases. However, this is true for systems that have been trained with massive amounts of data and have reached high levels of accuracy.
In this case, nurses were frustrated because the system rejected images (for quality) that showed no signs of retinal damage, but the nurses had no mechanism of intervening. They had to redo the process with the patient just to confirm the facts they already knew.
It sounds obvious, but the key to building a successful AI-application is to stress test it in a range of environments and create flexibility in workflows that can ensure high levels of productivity while preserving the integrity of the results.
How would you address these challenges? I'd love to hear your thoughts.
These insights were extracted from a recent article published in the MIT Technology Review.
Amit Rawal is a Sloan Fellow at Stanford's Graduate School of Business. He has spent the last decade in building and scaling e-commerce ventures for 40%+ of the world's population. At Stanford, he is focused on bringing together tech, design, and data to create joyful shopping experiences. He is a data geek and loves tracking all kinds of health and wellness metrics. He can be reached at [email protected].
Deploying Gen AI & ML for the greater good - Sales | Solutions | Strategy | Product
4 年Love it. Good succinct points without belabouring the topic. You may want to read up things that Stuart Russell has written up. https://people.eecs.berkeley.edu/~russell/ His book called Human Compatible is a mainstay in the AI field and is dealing exactly with the issue of how to build systems that look upon humans to question whether they need more context before trying every possible path to reach the specified motive. example: If you tell AI to drastically reduce CO2 levels as your main goal, they may attempt to contain human life or prevent more births to achieve that but that is not a good solution for us.
CPO: AI Stack | ? Apple AI Leader | Stanford | AI Professor
4 年Julia Gong thanks for sharing the piece that led to these learnings. Team: John Kamalu Nikita Namjoshi Susannah Shattuck - thanks for enriching my learning during our Google study. Dream team!