Unlocking the Potential of LLMs in Healthcare: A Case Study from Dana-Farber Cancer Institute

On September 6, 2024 Jason Johnson, PhD and Renato Umeton, PhD from Dana-Farber Cancer Institute gave a talk entitled "Unlocking the Potential of LLMs in Healthcare: A Case Study from Dana-Farber Cancer Institute". In 2023, Dana-Farber became the first academic medical center to deploy a LLM for general use. In this talk, Dr. Johnson and Dr. Umeton shared insights and lessons learned. They hope that these insights will be beneficial to other organizations considering similar deployments of AI as the industry grapples with the concurrent imperatives of innovation, cost-effectiveness, and patient safety. This talk was hosted by the AI Precision Health Institute at the University of Hawai'i Cancer Center as part of a popular seminar series featuring leading AI researchers from around the world.

In this talk, Dr. Johnson and Dr. Umeton shared their experience implementing LLMs in a private, secure, and HIPAA-compliant way.

• They explained how they adapted the GPT family of models (i.e., GPT-3.5 Turbo, GPT-4 Turbo, GPT-4o) to their specific needs and goals, and how they addressed some of the ethical, legal, and technical challenges.
• They discussed why they excluded direct clinical use of LLMs (i.e., to treat, diagnose, or drive/inform clinical management) and limited clinical explorations to clinical research studies and institutionally sanctioned IT pilots.
• They discussed ancillary activities required for broad AI deployment, like policy, ethics, training, monitoring, user support, and working with external companies for safe and impactful AI adoption.

Recording of the talk on Youtube: https://www.youtube.com/watch?v=9eDmFSdM4SE

"Clinical AI is both high-ROI and high-risk. By focusing on AI for research and operations we are pursuing use cases that are high-ROI and lower-risk."

Renato Umeton, PhD, Director, AI Operations and Data Science Services, Dana-Farber Cancer Institute

Highlights & Slides

Dana-Farber Deploys GPT-4 In 6 Months

Dana-Farber Cancer Institute, known for groundbreaking cancer research, is also an AI Healthcare pioneer. They are using GPT-4 to streamline work and conduct research, but not in direct clinical care. Although generative AI clinical pilot studies are taking place in many hospitals, Dana-Farber was the first to deploy a LLM for general use in an academic medical center or hospital. In March 2024, Dr. Johnson and Dr. Umeton and colleagues published a paper in NEJM AI sharing the approach, codebase, and lessons learned during the deployment to help others accelerate their AI programs. The paper entitled GPT-4 in a Cancer Center- Institute-Wide Deployment Challenges and Lessons Learned describes the use of GPT-4 in all business areas, including basic research, clinical research, and operations.

Highlights from the paper: https://www.dhirubhai.net/pulse/dana-farber-cancer-institute-deploys-gpt-4-margaretta-colangelo-wdzff/

Speakers

Jason Johnson, PhD

Chief Data and Analytics Officer Dana-Farber Cancer Institute

Jason Johnson is Chief Data and Analytics Officer and SVP of Informatics and Analytics at the Dana-Farber Cancer Institute in Boston, where he has served for 8 years. Jason’s team at DFCI includes research informatics, business intelligence, AI operations and data science services, bioinformatics and molecular data, data governance and architecture, data warehousing, informatics support for clinical trials, scientific computing, and software engineering. Prior to joining Dana-Farber, Jason was Head of R&D at PatientsLikeMe, a patient-focused research company in Cambridge, MA. He came to that position after 16 years in the biotech and pharmaceutical industries in various leadership roles in computational sciences, informatics, IT, and genomics. His last role at Merck & Co., Inc. was Associate VP, Scientific Informatics. Jason has undergraduate degrees in Philosophy and Physics from Stanford University, a Master’s degree in Physics from the University of Cambridge (UK), and a PhD in Biophysics from Harvard University.

Renato Umeton, PhD

Director of AI Operations and Data Science Services, Dana-Farber Cancer Institute

Renato studied computer science for both Master’s and Bachelor’s, later he earned a Ph.D. in Mathematics and Informatics defending a thesis on Optimization and Ontology for Computational and Systems Biology, which brought him to work first at Microsoft and then at MIT. Additionally, he has pursued Executive Education at Harvard to complement his existing leadership skills.

Currently Renato serves as Director of Artificial Intelligence Operations and Data Science Services in the Informatics & Analytics department of Dana-Farber Cancer Institute, a teaching affiliate of Harvard Medical School. In this position, where he reports to the Chief Data and Analytics Officer, Renato created the departmental AI & data science horizontal, counting about 40 people (including temps) as of 2024. He accrued 15 years of experience in artificial intelligence, data science, and big data working across hospitals, academia, consulting, and in industry, where he operated in roles spanning from postdoc to director. In those contexts, he worked on several scientific publications and patents, some of which were leveraged in clinical trials or licensed. As of 2024, Renato co-authored 140+ scientific publications, 6+ patent applications, and he is currently affiliated also with MIT, Harvard School of Public Health, and Weill Cornell Medicine. Renato also participates in various data science academic-industry-government collaborations that aim at democratizing and expanding the reach of artificial intelligence and machine learning in healthcare, regulatory science, and healthcare financing.

In addition to his main responsibilities, he is a reviewer for various journals by NEJM Group, Nature Publishing Group, Cell Press, IEEE, ACM, and Oxford Press among the others, (b) invited speaker at top venues (e.g., US Federal Government Departments; Fortune 20 companies; top academic medical centers), (c) led several ML and AI communities and conferences, (d) mentor to 50+ mentees through programs varying from bootcamp to MD-PhD degree, and (e) manager for an online ML community counting 80,000+ members world-wide.

AI Precision Health Institute Seminar Series

In 2022 we formed the AI Precision Health Institute Affinity Group Seminar Series to discuss current trends and applications of AI in cancer research and clinical practice. The group brings together AI researchers in a variety of fields including computer science, engineering, nutrition, epidemiology, and radiology with clinicians and advocates. The goal is to foster collaborative interactions to solve problems in cancer that were thought to be unsolvable a decade ago before the broad use of deep learning and AI in medicine.

Past Seminars In This Seminar Series

Dynamic Prediction Improved Breast Cancer Risk Prediction, August 2024

Large Language Models For Biomedical Research, July 2024

What Happens If We Use Synthetic Data Without Any Curation, June 2024

Predictive AI Models - Data Standards In Action, May 2024

AI Powered Dermatology Tools and Consumer Decision Making, April 2024

AI Decodes Waveforms To Help Prevent Sudden Cardiac Death, March 2024

Mitigating Unintended Consequences of AI in Biomedicine, February 2024

How To Build Responsible, Safe, Trusted AI For Precision Health, January 2024

Robust Interpretability Methods For Large Language Models, December 2023

Machine Learning Captures Insights Into Brain Tumor Biology, November 2023

Comparing AI Algorithms To Predict 5 Year Breast Cancer Risk, October 2023

Disrupting the Indigenous DNA SupplyChain, September 2023

AI Based Lab Test Approved To Phenotype, Grade Breast Cancer, July 2023

Trustworthy AI and Clinical Validation In Breast Cancer Imaging, June 2023

AI For Ultrasound For Real-Time Breast Cancer Decision Support, May 2023

Deep Learning To Diagnose Breast Cancer With High Accuracy, April 2023

Precision Oncology: Empowering Radiologists With AI, January 2023

Machine Learning For Personalized Cancer Screening, December 2022

AI Driven Surgical Robots To Diagnose/Treat Prostate Cancer, November 2022

Subscribe, Comment, Join Group

I'm interested in your feedback - please leave your comments.

To subscribe to the AI in Healthcare Milestones newsletter click here .

To join the AI in Healthcare Milestones Group click here .

Copyright ? 2024 Margaretta Colangelo. All Rights Reserved.

This article was written by Margaretta Colangelo. Margaretta is a leading AI analyst who tracks significant milestones in AI in healthcare. She consults with AI healthcare companies and writes about some of the companies she consults with. Margaretta serves on the advisory board of the AI Precision Health Institute at the University of Hawai?i?Cancer Center @realmargaretta