Synergized LLMs + Graphs
Prologue
Since ChatGPT's launch in November 2022, Large Language Models (LLMs) have captured the overwhelming attention of the global public. LLMs have exhibited human-like, and in some cases, exceptional performances on a vast spectrum of natural language tasks. Despite various ethical and societal concerns, the world is not reversing this trend.
However, putting aside the technological enthusiasm, how can enterprises get substantial advantages from LLMs, aside from the advantages that individual employees naturally gain through their daily usage, enhancing productivity?
Sufferings of LLMs
Advanced LLMs, such as GPT series models, have a dramatical size of billions of parameters, exhibiting great potential in many complex tasks. They undergo extensive pre-training on vast amount of text data, the process is notably resource-intensive and time-consuming.
It’s easy to understand that the model’s knowledge is restricted to the data it was trained on, which has a cutoff date. For instance, GPT-3.5 was trained with corpora sourced till September 2021, and GPT-4 is currently up to January 2022. Keeping up with real-time knowledge and emerging global events poses a significant challenge for LLMs and comes at a considerable cost.
Worse, rather than admitting that what you asked is not in their knowledge base, LLMs might give you responses sound affirmative but are not the facts, leading to what is known as hallucinations. Meanwhile, as black-box models, LLMs represent knowledge implicitly in parameters. There is no source or reference given along with the responses, so it’s hard to interpret or validate the credibility.
This severely impairs the organizational application of LLMs, especially for high-stakes contexts like medical diagnosis, financial advice and legal judgement. The challenges also arise from the fact that public LLMs are trained for general purposes and not aware of your organization’s proprietary, confidential, or sensitive data that has never been made public.
A simple approach to improve the performance of LLMs is prompt engineering. By providing clear instructions and context alongside your question, LLMs can generate more accurate responses. However, this process may require creative writing skills and multiple iterations to obtain the best possible response, with the associated increase in costs due to longer text tokens.
Demystifying: RAG and Fine-Tuning
Two notable techniques have emerged in localizing LLMs into business environment: RAG (Retrieval Augmented Generation) and Fine-Tuning. These two approaches involve distinct implementations and costs.
RAG applications incorporate a retrieval system to fetch relevant documents from an org-specific database. These documents are then passed along with the user query to the pre-trained LLM to generate the response.
The typical RAG strategy involves segmenting the org-specific documents into shorter, meaningful chunks. Each of these chunks is embedded into a vector. During operation, the system generates a vector embedding for the user input and retrieves the top N (e.g., 3) similar text chunks by comparing their vectors to the input vector. Subsequently, the LLM generates a response based on the user input and the relevant documents.
Fine-tuning starts with a pre-trained LLM as the base, and further trains it with a specific dataset to adjust its parameters. The dataset used for fine-tuning is generally smaller than what’s required for pre-training, but it still demands significant efforts and resources, and it must align with the specific task. In the end, you will have a refined LLM which performs better than the base LLM in specific task like question-answering, translation, and sentimental analysis.
Both RAG and fine-tuning offer solutions to mitigate the knowledge cutoff and hallucination issues associated with pre-trained LLMs, making LLMs more relevant and current to your organization. The choice between RAG and fine-tuning hinges on the specific task, and there are approaches that combine these techniques. In any case, you can’t leave graph behind in order to unlock the full potential of LLMs!
Graph Abilities You Want in LLMs
A more pronounced limitation of LLMs shows up when tackling deep reasoning and relating tasks. When we dare to ask bold questions, we often encounter intriguing scenarios. For instance, Ricky Sun, the CEO of Ultipa, was once inspired by a radio discussion that posed a captivating question: What is the connection between the renowned Eastern conqueror, Genghis Khan, and the preeminent Western scientist, Isaac Newton?
ChatGPT falls short in answering this question, because they “lived in different times and places and had entirely different backgrounds and roles in history”.
In contrast, if you explore it in a knowledge graph populated with encyclopedic data (you can do this with Wikipedia), many results can be found when searching for paths between Genghis Khan (node) and Isaac Newton (node). Here is one of them:
Genghis Khan launched the Mongol Invasion of the West, which brought about the outbreak of the Black Death in Europe. The plague ravaged Europe for centuries, affecting its development. During the London Plague outbreak, it was Newton's era, and at that time, he was discharged from college due to school shutdown, and in the countryside, avoiding the plague.
This 4-hop causal connection between Genghis Khan and Newton spans 400 years across East and West, and it’s vividly presented through graph penetration (deep traversal and causality search).
A wide range of real-world data can be modeled as graphs, encompassing financial networks, transportation networks, supply-chain networks, social networks, biological networks and all kinds of knowledge graphs. Graphs are renowned for their interconnected, traceable, and explicit ways of representations. You can reframe many problems as graph problems:
These analytical and algorithmic questions often transcend the scope of typical text-based queries. While current LLMs may excel in some areas, they tend to struggle with basic mathematical questions, as illustrated below when inquiring about the shortest route.
LLMs excel in their designated task of generating coherent text in response to input. Despite demonstrating some emergent abilities, including math problem-solving, their current performance lacks reliability. Graphs, in contrast, are grounded in the solid foundation of graph theories and offer explicit analytical and algorithmic frameworks, making them dependable problem solvers with well-defined responsibilities.
领英推荐
Hybrid Solution: LLMs + Graphs
Many researchers have discerned the inherent complementarity of LLMs and graphs. By leveraging the textual comprehension abilities of LLMs and the structured reasoning power of graphs, their unification presents compelling opportunities to augment the capabilities, intelligence, and interpretability of AI systems.
In this section, we will discuss two applications Ultipa has implemented to demonstrate the mutual reinforcements between LLMs and graphs.
LLMs for Graph Construction
The process of transforming structured data, such as tabular data, into graph is relatively straightforward. By designing an appropriate graph model, including schemas and properties for nodes and edges, one can easily construct a graph with explicit mappings and migrations.
Yet, creating graphs from raw text data is complicated that requires advanced NLP techniques like Named Entity Recognition, Relation Extraction, and Semantic Parsing. However, these methods may require extensive manual rule-based or supervised training, and their performance can vary depending on the quality and quantity of training data.
LLMs, on the other hand, have demonstrated remarkable performance in addressing the entity discovery, coreference resolution, and relation extraction from large-scale unstructured text data. They present exciting opportunities to automate and expedite the graph construction process.
The shortcut [1] Graph Extractor is designed to extract information of entities and relations in the user input text and directly visualize them in a graph. Below is an overview of its workflow.
Specifically, the following prompt was developed to instruct the LLM (GPT-4) to extract information from the given text in a certain format:
{
"role": "system",
"content": "Extract entities and relations from the given text."
},
{
"role": "user",
"content": "Return results in this format: (Entity) - [Relation] -> (Entity)
If the name of the entity or relation is long or informal, transform or summarize it. Keep the same name if the entity or relation appears multiple times in different formats.
Example Text: Apollo 11 was launched on July 16, 1969, at 8:32 AM Central Daylight Time (CDT) with the goal of performing the first human landing on the Moon. Commander Neil Armstrong, Command Module Pilot Michael Collins, and Lunar Module Pilot Edwin “Buzz” Aldrin entered lunar orbit on the afternoon of July 19. The following day, Armstrong and Aldrin begin their descent to the lunar surface in the Lunar Module, Eagle. The planned landing site in the Sea of Tranquility was selected as a flat, safe location and had been surveyed by Apollo 10 at an elevation of 10 miles above the Moon. However, a navigation error earlier in the mission caused Eagle to be about 7 kilometers beyond the planned landing location.
Results:
(Apollo 11) - [launched on] -> (July 16, 1969)
(Apollo 11) - [had goal of] -> (the first human landing on the Moon)
(Apollo 11) - [had] -> (Lunar Module)
(Lunar Module) - [is named] -> (Eagle)
(Apollo 11) - [had] -> (Command Module)
(Neil Armstrong) - [is Commander of] -> (Apollo 11)
(Michael Collins) - [is Pilot of] -> (Command Module)
(Edwin Aldrin) - [is Pilot of] -> (Lunar Module)
(Apollo 11) - [entered] -> (Lunar orbit)
(Eagle) - [descent to] -> (Lunar surface)
(Neil Armstrong) - [descent to] -> (Lunar surface)
(Edwin Aldrin) - [descent to] -> (Lunar surface)
(Eagle) - [planned landing location] -> (Sea of Tranquility)
(Lunar surface) - [has a flat, safe location] -> (Sea of Tranquility)
(Sea of Tranquility) - [was surveyed by] -> (Apollo 10)
(Eagle) - [actual landing location] -> (about 7 kilometers beyond)"
},
Testing Example: We tested its performance by giving it an article about the genetic link between Synesthesia and Autism:
Upon reviewing the results against the source text, it appears that the information extracted by the LLM is largely accurate. However, there is an issue with the graph structure. Although the graph is relatively small, it is disconnected into 7 components, which could lead to breaks in the flow of information. For example, the core finding represented as "(Synesthesia) - [shares genetic roots with] -> (Autism)" is isolated from related research like "(Researchers) - [examined] -> (2,131 pairs of twins)".
To address this issue, it would be beneficial to fine-tune or instruct the LLM with some standard entity and relation types or schemas. For instance, (@discovery) – [@contains] -> (@phenomenon), and (@discovery) – [supportedBy] -> (@research). This approach can improve the overall structure and readability of the graph, ensuring that related information is appropriately linked and presented.
Graph Question-Answering
Graph question answering aims to find answers to natural language questions based on the structured facts stored in graphs. Harnessing LLMs proves highly effective in bridging the gap between these inquiries and the underlying graph data.
The shortcut [1] ChatGraph empowers users to engage with their data conversationally. Given the proficiency in language comprehension, LLMs are effectively utilized for understanding and identifying information mentioned in natural language questions and translating the questions into the precise graph queries or algorithms in UQL (Ultipa Query Language).
Testing Example: To illustrate this prowess, consider a scenario in a power grid network. We initiated a query: "How can we connect KANAWHA and TAZEWELL?" The system promptly retrieved all viable pathways connecting these two locations using the AB path query:
Additionally, we posed another question: "What is the most efficient way to reach TAZEWELL from KANAWHA?" The system recognized this as a shortest path problem and promptly provided the optimal solution:
Looking into the Future
In the ever-expansive realm of information technology, two powerful catalysts have emerged, fundamentally altering our approach to data processing and analysis: Large Language Models (LLMs) and Graph Databases. While LLMs have showcased remarkable capabilities in understanding and generating natural language, the concerns and criticisms around them have also cast shadows on their applicability in serious business contexts.?
Nevertheless, the synergy between LLMs and graphs holds the promise of mitigating the limitations associated with LLMs. By harnessing the explicit and structured representation of relationships provided by graphs, this harmonious integration anticipates a forthcoming era in data processing and problem-solving. In this envisioned future, the inherent strengths of both technologies will collaborate to overcome existing challenges and obstacles.
Authors: Pearl Cao, Jason Zhang
Click here to download the PDF version of this ariticle.