LLM Applications in HealthCare

NLP Summit 10/2023 write-up pt. 2/2

This is part two of my write-up of the John Snow Labs NLP Summit from Oct. 3rd-5th 2023. It focuses on NLP applications in healthcare, while part one was about general trends and technologies.

Executive Summary

Most applications for ML in HealthCare are still in early stages of development. Medical Imaging is the notable exception - but this is not due to LLMs. Data availability and standardization remain challenging and knowledge extraction is primarily done with "traditional" NLP methods like Named Entity Recognition (NER). Most of these issues aren't new and even the biggest players (like Google) have them. Clinical language models still outperform general-use LLMs for reasoning tasks, therefore use of LLMs is mostly limited to user interfaces at the moment (think: "medical chatbots" and "chat with your documents"). Multimodal LLMs might change this outlook, but as of now, there is simply no "killer app" using LLMs in HealthCare.

Multi-modality is always right around the corner

One school of thought argues, that models need multi-modal inputs to be successful in medical reasoning. Intuitively this makes sense - after all, a physician also takes several modalities (e.g. Text, Images, Sensor-Data) and their changes over time into account, when diagnosing and treating patients. Both Durga S. Borkar, MD, MMCi hinted at this, and Zain Hasan gave an introduction on how this could be facilitated with Vector-DBs.

While the concept of multi-modality isn't specific to LLMs (see slide below), this is one development to watch out for in the next 12 months with recent announcements from OpenAI.

Cohort Definition / Precision Medicine

Shaayaan Sayed from ClosedLoop demonstrated their natural language interface to their risk stratification platform. Users can ask questions like "show all patients that are at high risk to be admitted to the ED in the next 6 weeks" and the LLM generates custom SQL queries to retrieve this patient cohort. While this sounds straight forward, they also struggled with input size limitations and had to couple their inference models with additional classifiers to reduce the number of columns that would be relevant to such a generated query. So context length limits strike again (see pt. 1).

Dr. Richard "Spencer" Schaefer from the US VA showed their initial results on their quest to extract knowledge from large amounts of Electronic Medical Records (EMR) by using LLMs to query and summarize findings to the users specific questions. Unfortunately he wasn't at liberty to discuss detailed results. From their evaluation of self-hosted models, they seem to lean toward Llama-2 for their domain-specific tasks. However, the project still seems to be in early stage, maybe they'll release a more detailed report in the future.

Knowledge Extraction

Itay Zitvar and Yizhen Zhong both reported on their experiences with using GPT architectures / LLMs for domain-specific NLP tasks and noted that they underperformed compared with hybrid approaches or BERT models.

Where will all the medical text for training come from?

De-Identification is at the top of my current interests, because high quality de-id is always the first step to enable knowledge extraction for unstructured text. In relation to our recent poster on using on-premise LLMs for de-id (find the abstract here), David Talby from John Snow Labs showed some interesting numbers and facts that really question the economic feasibility of using LLMs for de-identification tasks:

While ETL and de-identification isn't specific to medical applications, there was one talk that really ticked a lot of boxes for me in terms of daily needs and pain points. Matt Robinson from unstructured.io demoed their ETL library. I think this can be really valuable to speed up POC / MVP development and I will probably try their open-source lib in the future, and see how it scales from there.