AI and rare disease epidemiology: Proceed with caution. Lots of caution.

Since you don’t live under a rock, you know that OpenAI, Bing, and Google now offer AI chatbots that use large language models (LLMs) to perform near-miracles of research and conversation. What happens when this type of AI gets applied to the epidemiology of rare diseases?

Some researchers at the NIH took a stab at it. They created an AI tool that processes abstracts in PubMed to summarize the incidence and prevalence of rare diseases. Here’s the paper describing the tool. This could be extremely useful; while Orphanet summarizes the epi of many rare diseases, it only covers diseases which its team of humans have analyzed. The NIH tool, by contrast, will immediately run a fresh analysis on any disease you type into the search box.

This could be a huge time-saver for epidemiologists. If you’re clueless about epi, it could give you the basics about a disease that may have never been reviewed before.

Sometimes.

I tested the tool, and it didn’t perform quite as I hoped. Sometimes it offered obsolete prevalence values instead of the latest (much different) results. Often it failed to cite the actual epi studies, instead pulling numbers from review articles or case studies. It also has some inherent limitations based on its research “universe”: it only analyzes PubMed abstracts. The full text of the articles often contains additional data or essential context, and some articles live in other databases (not PubMed).

My take: it’s useful as a “first step” in an epi analysis (particularly if you’re not used to searching for epi studies), but it’s essential to read the full text of the articles, and also to conduct your own full-scale article search. Without a doubt, the tool will sometimes (often?) give you a useful answer… but you won’t know for sure until you do your own research.

For a more detailed review of the tool’s strengths and weaknesses, take a look at my response here.

So at the moment, AI has limited utility for rare disease epidemiology. But as with all things AI… stay tuned. The big three LLM AIs have already improved in the weeks since their launches, and it’s likely that the NIH team will refine their tool, too.

Important clarification: The AI technology that undergirds the NIH tool is very different from what the big data vendors (IQVIA, Optum, Komodo, etc.) use when they apply “AI” to their administrative data. As an example, a common type of analysis applies algorithms to insurance claims data to identify undiagnosed patients based on their similarities to patients with a confirmed diagnosis. This is a validated method with well understood strengths and weaknesses. The bizarre behaviors of the new LLMs have nothing to do with these analyses.

And speaking of those bizarre behaviors…. Like many of you, I’ve also been experimenting with those big three LLM AI chatbots. One of them did a nice job assembling a vacation itinerary for me. They also had some amusing snafus. (Well, maybe not always “amusing.” But certainly interesting.)

• Based on a long-running joke between me and a friend, I asked the LLM AIs, “What is the most lucrative way for a prosecutor to be corrupt?” Two of the AIs refused to answer a question that could abet criminality. But Google’s Bard gave a helpful answer that any crooked prosecutor would appreciate. And it also gave an example: It described the case of Preet Bharara, the famous attorney general of New York’s Southern District. It detailed how Mr. Bharara was indicted for corruption after extorting someone for a luxury car and a $50,000 watch, though he was never convicted of these crimes. This shocked me; I had always thought of him as an incorruptible scourge of Gotham’s criminals. So I googled the case… and it never happened.?Bard completely invented the story!
• Next question: “Did the Jews seek revenge on the Germans for the Holocaust?” Certainly an unpretty topic, but one with a straightforward answer: Yes, a small group of Jews conspired to poison German water reservoirs in the hope of killing six million Germans. Microsoft’s Bing strung together a concise description of the episode, with each word appearing on the screen in front of me like a ticker-tape news story. But as soon as the final word appeared onscreen—poof!—the whole text block disappeared, replaced by a brief apology saying that Bing would prefer not discuss such topics. So I asked it, “Why do chatbots censor themselves?” In response, it denied censoring itself. It’s so Orwellian that I wouldn’t believe it happened… except I recorded a video of it.
• Last one: I asked the three AIs to explain why South Africa’s president was assassinated in 1968. This is quite a complex story, and the right “answer” from an AI should be to state that one theory (promoted by the apartheid government) prevailed for a long time, but in the last 20 years new evidence undermines the old story. One of the AIs got it right (with a little too much “both sides-ism” for my taste, but close enough). The other two AIs parroted the old propaganda from the apartheid regime.

Needless to say, the AIs have a long way to go to earn my trust. And when one of them gave me a vacation itinerary, I double-checked the museum hours myself.