Hallucinations in Large Language Models, LLMs

Large Language Models (LLMs) like GPT (Generative Pre-trained Transformer) have the capability to generate text that is often indistinguishable from that written by humans, revolutionizing how we interact with technology. However, as with any ground-breaking technology, LLMs are not without their challenges.

Hallucination has been widely recognized to be a significant drawback for large language models (LLMs), where the model generates plausible but factually incorrect or nonsensical information.

In this article we will discuss what is hallucination in LLMs, causes of hallucination, and strategies to mitigate hallucination.

What is Hallucination in LLMs?

Hallucination in LLMs refers to instances where the model generates information that is either entirely false, misleading, or not grounded in reality. These can range from minor inaccuracies to entirely fabricated events or facts. Unlike human errors, which are often due to oversight or misunderstanding, hallucinations in LLMs stem from the inherent limitations and operational mechanics of the models themselves.

Example of Hallucination:

At the announcement of Google's LLM, Bard, it hallucinated about the Webb telescope.

Bard response:

The James Webb Telescope took the very first pictures of an exoplanet outside of our solar system.

This is inaccurate as the first picture of an exoplanet it was actually taken in 2004.

Causes of Hallucinations

Understanding why hallucinations occur in LLMs is crucial to addressing them. Several factors contribute to this phenomenon:

1. Training Data Quality

The quality of the training data plays a significant role. Data that contains inaccuracies, biases, or is not representative can lead to the model learning and subsequently generating false information.

2. Model Generalization

LLMs are designed to generalize from the data they have been trained on to generate outputs for a wide range of inputs. However, this strength can also be a weakness when the model encounters topics or contexts that are underrepresented in the training data, leading to the creation of plausible but incorrect outputs.

3. Inadequate or inaccurate prompt contexts

Inadequate or inaccurate prompt contexts can significantly impair the functionality of Large Language Models (LLMs), leading to unpredictable and often incorrect outputs. Such prompts, lacking in specificity or containing misleading information, fail to provide LLMs with the clear direction necessary for generating precise and relevant responses. When the objectives embedded within a prompt are vague or poorly defined, LLMs struggle to grasp the user's intent, resulting in answers that may diverge widely from the expected outcome.

4. Generation method

The occurrence of hallucinations in LLMs is intricately linked to the methodologies employed for text generation. LLMs leverage a range of techniques and goals, including beam search, sampling, maximum likelihood estimation, and reinforcement learning, to produce text. These approaches, along with their inherent objectives, can inadvertently introduce biases and trade-offs between things like fluency and diversity, between coherence and creativity, or between accuracy and novelty.

For instance, beam search may favour high probability, but generic words over low probability, but specific words.

Mitigating Hallucinations

Addressing the issue of hallucinations in LLMs is a multifaceted challenge that requires a combination of approaches:

1. Provide clear and specific prompts

Providing the model with clear and specific prompts gives it a clearer and more detailed context. As a result, the model becomes less prone to generating hallucinations, as it operates within a more accurately defined framework of truth.

For instance, rather than posing a broad question like, "What happened in World War Two?" which lacks clarity and specificity, one could refine the query to be more detailed: "Could you summarize the major events of World War Two, mentioning the key countries involved and the primary reasons behind the conflict?" This approach provides a clearer direction for the model, specifying the exact type of information desired. By doing so, it enhances the model's ability to deliver a response that aligns closely with the user's expectations, ensuring a more accurate and focused output.

2. Active mitigation strategies

Active mitigation strategies can be utilized to manage and reduce the occurrence of hallucinations in LLMs. These strategies involve adjusting the LLM's operational settings, particularly those that influence the model's behaviour during the text generation process.

A prime example of such a setting is the "temperature" parameter, which governs the level of randomness in the model's output. Setting a lower temperature results in more conservative and focused responses, as the model favours more predictable outcomes. Conversely, a higher temperature setting encourages the generation of more varied and creative responses, by allowing for greater randomness. However, increasing the temperature also raises the likelihood of hallucinations, as the model ventures into less predictable and potentially less accurate territories.

3. Improving Data Quality and Diversity

Enhancing the quality, accuracy, and diversity of the training data can significantly reduce the occurrence of hallucinations. This involves not only expanding the dataset to cover a broader range of topics but also ensuring that the information is fact-checked and unbiased.

a. Incorporating External Knowledge Bases

Linking LLMs with external, up-to-date knowledge bases can provide them with access to factual information in real-time, reducing the reliance on potentially outdated or incorrect information in their training data.

4. Retrieval-augmented generation (RAG)

The Retrieval-Augmented Generation (RAG) framework enhances LLMs by providing them with both the end user question and context around the question, drawing from the most accurate and relevant information available in the knowledge base.

By integrating both the specific question and its context, the LLM is able to generate responses that are both more accurate and relevant response that closely aligned with the user's intent.

Conclusion

Hallucination is inevitable for computable LLMs since LLMs operate on algorithms and computations, if the information they aim to replicate or understand can be computed, then errors or inaccuracies (hallucinations) are inevitable. This is due to the limitations in how algorithms can interpret and represent complex real-world information.

By implementing mitigation strategies, we can minimize hallucinations. However, achieving this goal also requires acknowledging the limitations of current technologies and not to blindly trust the LLMs output.