Poisoning Medical Knowledge – Can Large Language Models Disrupt Science?

In recent years, large language models (LLMs) like ChatGPT have emerged as powerful tools, capable of generating high-quality text on nearly any topic. However, they also bring significant challenges, especially in sensitive systems like medical knowledge.

Imagine a scenario where medical research is based on a knowledge graph (KG)—a system connecting drugs to diseases, supported by text analysis from hundreds of thousands of research papers. These connections can validate existing discoveries and generate new research hypotheses, accelerating scientific progress. But what happens when such a graph is contaminated with false information?

A new study from the University of Washington and Peking University reveals a concerning phenomenon: using an advanced model named Scorpius, researchers were able to introduce fake papers into a medical knowledge graph and manipulate its outcomes. Leveraging ChatGPT and BioBART, Scorpius can create abstracts that promote a specific drug and suggest a misleading association between the drug and a particular disease. This study found that adding just one malicious abstract could boost a drug’s ranking to the top of the graph, potentially misleading researchers or physicians into believing the connection is legitimate.

Ethical Implications and Accountability in the Information Era

These findings raise substantial ethical questions. On one hand, they serve as a warning to use AI tools judiciously, particularly in sensitive fields like medicine. On the other, they highlight the importance of rigorous scrutiny for information sourced from unreviewed outlets, like preprint repositories. As LLMs improve, the challenge of distinguishing between true and false information will only grow.

What Can Be Done?

The scientific community and technology companies must collaborate to develop better tools for detecting fake content, and they might even consider restricting the use of LLMs for generating medical content. Such initiatives could ensure that medical information remains trustworthy, minimizing the risks of disruption.

In Conclusion

As models like Scorpius show the potential to inject misinformation into critical systems, our responsibility is to ensure these technologies uphold ethical standards and serve us, not harm us. AI can accelerate discoveries and contribute to a better world—but only if we use it responsibly and with care.