Using Chat-GPT to Generate Structured Biological Knowledge

After my previous post on using Chat-GPT to explain biological findings, I was interested in digging in a bit more.? Specifically, I wanted to explore the reverse idea - rather than using Chat-GPT to provide interpretation on structured data, could I flip things around and use the tool to generate core lists for further analysis?

Anyone who has been in informatics at all is familiar with some of the vendor-provided tools like Qiagen’s Ingenuity IPA or Clarivate's Metacore. These are great tools for pathway analysis and interpretation, but rely on a core database of curated discoveries from the literature.? Protein A binds protein B, protein B phosphorylates protein C, and so on.? Text mining approaches are sometimes used to supplement these (or in some tools, replace them entirely), but I was wondering how Chat-GPT, a fairly general tool, could perform at these tasks.

The results were…mixed.? This was a case where I’d found that the OpenAI API models did not do nearly as well as Chat-GPT.? Sometimes they would recover ligands for a receptor, sometimes they would just dump out lists of similar receptors.? I found that Chat-GPT itself was more convincing (note that original output was a table, but that was not supported in this format of post):

What are proteins that bind to the protein CCR1??

• Entrez Gene; Gene Symbol; Relation Type
• CXCL12; CXCL12; Ligand
• CCL5; CCL5; Ligand
• CCL3; CCL3; Ligand
• CCL4; CCL4; Ligand
• CXCL4; PF4; Ligand
• CXCL8; IL8; Ligand
• CCL15; CCL15; Ligand
• CCR1; CCR1; Homodimerization
• G protein subunits; GNAI2, GNAI3, GNAQ; Signal transduction
• β-Arrestin; ARRB1, ARRB2; Signal transduction

That is a nice result (leaving aside the headers of "entrez gene and gene symbol" not being 100% accurate)…but without an API, I’m not exactly going to build my own interactome for all the protein-coding genes.? Worth revisiting once API-access to Chat-GPT becomes available.

Emboldened by this, I attempted another query - what if I could build a directory of disease-relevant animal models?

What are specific mouse models for multiple sclerosis, displayed as a table with columns for the model name, phenotype, and the reference?

• Model Name; Phenotype; Reference
• Experimental autoimmune encephalomyelitis (EAE); Paralysis, demyelination, and inflammation in the central nervous system; Stromnes and Goverman, 2006 (PMID: 16815317)
• Theiler's murine encephalomyelitis virus-induced demyelinating disease (TMEV-IDD); Demyelination and inflammation in the central nervous system; Lipton and Miller, 2005 (PMID: 15872086)
• Cuprizone-induced demyelination model; Demyelination in the central nervous system; Matsushima and Morellini, 2001 (PMID: 11676928)
• Chronic relapsing experimental autoimmune encephalomyelitis (CREAE); Relapsing-remitting paralysis, demyelination, and inflammation in the central nervous system; Lyons et al., 1999 (PMID: 10207173)
• Spontaneous autoimmune encephalomyelitis (S-AE); Paralysis, demyelination, and inflammation in the central nervous system; Huseby et al., 2001 (PMID: 11514607)
• Myelin oligodendrocyte glycoprotein (MOG)-induced experimental autoimmune encephalomyelitis (EAE); Paralysis, demyelination, and inflammation in the central nervous system; Bettelli et al., 2003 (PMID: 12914783)
• Proteolipid protein (PLP)-induced experimental autoimmune encephalomyelitis (EAE); Paralysis, demyelination, and inflammation in the central nervous system; Sun et al., 2001 (PMID: 11356892)
• T cell receptor transgenic (TCR-Tg) mice; Paralysis, demyelination, and inflammation in the central nervous system; Anderson et al., 2003 (PMID: 12829604)
• Inducible costimulator (ICOS)-deficient mice; Reduced severity of paralysis, demyelination, and inflammation in the central nervous system; McAdam et al., 2001 (PMID: 11546870)
• Interleukin-7 receptor alpha (IL-7Rα)-deficient mice; Reduced number of T cells and less severe paralysis, demyelination, and inflammation in the central nervous system; Peschon et al., 1994 (PMID: 8001160)

Incredible! A list of models, some entirely new to me (although I don’t work on MS as an indication, so maybe that isn’t a surprise…), along with a rough description of phenotype that could allow for a good selection, and a reference.

But wait…what if we actually have a look at one of these references?? Perschon et al did actually write a paper in 1994 about IL-7Ra deficient mice. But it is cataloged under PMID 7964471.? The PMID referenced in the table is a totally different paper from 1994, that has some connection with neurons, but is not the reference that was presented.? The ICOS model’s reference is even farther off - the PMID points to paper analyzing the size of U.S. businesses.?

Maybe it is not surprising that models trained largely on narrative text do a bit better with extracting descriptions than some of these references, but it’s definitely a cause for caution. The kinds of errors that can creep in with Chat-GPT look very convincing.? I think for the immediate future, for scientific work, Chat-GPT might work more effectively when primed directly with the relevant facts (where it can then reformat them), rather than direct question answering - unless the asker is prepared to take caution in confirming the results.