On to Knowledge-infused Language Models

A broad and deep body of on-going research? – hundreds of experiments! – has shown quite conclusively that knowledge graphs are essential to guide, complement, and enrich LLMs in systematic ways. The very wide variety of tests over domains and possible combinations of KGs and LLMs attests to the robustness of this foundational relationship.?

Several excellent and up-to-date studies (like those below) have documented carefully this research and shown the many ways in which

Knowledge graphs and LLMs together can push AI forward to create novel models that have knowledge infused into every single step of their development, evaluation, and deployment.

• Pan, J. et al. Large Language Models and Knowledge Graphs (2023)
• Pan, S. et al. Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023)
• Wei, et al. Knowledge Enhanced Pre-trained Language Models: A Comprehensive Survey (2021)
• Yan, et al. ChatGPT is not Enough: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling (2023)
• Zhen, et al. A Survey on Knowledge-enhanced Pre-trained Language Models (2023)

These studies make it clear that piling on ever more data for building even larger language models, enabling longer contexts, and developing more and more sophisticated prompt "engineering" pipelines are nowhere near as effective as addressing domain-specific reliability head on with structured knowledge. As we've seen time and time again, quantity is no guarantee of quality.

Knowledge graphs compress the vast linguistic variability of thousands of paraphrase sentences into a handful of standardized and human-readable triples. This makes knowledge graphs almost scale invariant with respect to sentences – the number of triples grows very slowly while the number of sentences covered by the knowledge graph can grow exponentially or faster. Since they compress so much information, triples are rocket fuel for training models, in one apt phrase. Moreover, many applications benefit significantly even from small-scale knowledge graphs or ontologies that include only hundreds or thousands of concept nodes. These two characteristics render expert human curation viable with respect to both cost and time – even if we work, as we do today, without the effective tooling that would enhance human productivity by a huge margin. Curation, in turn, plays a crucial role in early-stage technologies to build both the reliability and trust that drive widespread adoption and investment. Curated triples accumulate and this accretion documents both progress and return on investment -- they don't have to be re-generated and re-reviewed at each iteration. Moreover, the transparency enabled by curatable knowledge structures like knowledge graphs also goes a long way toward reducing the already extreme inefficiency of using trial and error for development, tuning, prompt "engineering", evaluation, and troubleshooting -- one hallmark of building today's LLMs – and the resulting unclear ROI is a widely-cited blocker to wider adoption.?

Common objections to knowledge graphs are most often not well founded:?

• Knowledge graphs do not have to encompass the whole universe of knowledge but can add value by including only the most important domain-specific concepts.??
• Knowledge graphs do not have to be built entirely by hand but can be automated to a significant extent.??
• Knowledge graphs do not have to be built by the engineers who will consume and deploy them, but can (and should) be built by equally expert and less costly analytic linguists and ontologists.?
• Knowledge graphs do not have to be discarded after ingesting into LLMs, but need to be retained to enable additional curation, troubleshooting, and compliance.?
• Knowledge graphs do not have to be discrete – used for lookup in discrete relational databases – they can be re-indexed as continuous embeddings and stored in vector databases or used to identify shortest paths and fuzzy matches over variable neighborhoods, in graph databases.?

Large language models, on the other hand, virtually revel in linguistic variability – this is where they shine and add the most value.? There's no doubt they constitute a quantum leap in natural language processing functionality.? Even the smaller, "nimble" language models that are cheaper, more flexible, and easier to build have enabled useful applications for the more restricted domains that dominate use cases in practice – improving both ROI and time to market.? But accuracy as well as access to and incorporation of reliable, trustworthy domain-specific knowledge remain key stumbling blocks to broader adoption of LLMs.??

There is some -- but still not enough -- work on leveraging the superpowers of LLMs to deal with the challenges of linguistic variability so we can build knowledge graphs and ontologies not automatically (and unreliably) but more effectively with punctual, verifiable curation.? Many studies show this to be a promising application of LLMs:? to compress extremely variable linguistic data into curatable triples.

The broad array of studies in these reviews support the view that

We are expecting far too much to think that LLMs might be able to reliably model reasoning and knowledge on their own.

It is already clear that LLMs need additional resources to move beyond modeling strings, as they do so well.? I like to call LLMs "the ultimate API for humans". It seems straightforward to conclude that leveraging them for this key strength and developing other components for reasoning and knowledge will yield better systems faster. Multimodal language models in which one "modality" is text and another is explicit knowledge are one important and promising way to make this happen with existing resources and algorithms.

Merging knowledge graphs and LLMs in the many ways explored in these experiments will take us beyond merely "knowledge-aware" models to models that have knowledge infused into every single step of their development, evaluation, and deployment.? This symbiotic KG + LLM approach will allow us to systematically leverage the strengths of each and make each support the other.