Detecting model hallucinations in Retrieval Augmented Generation (RAG) AI systems

Model hallucination, also known as AI hallucination, is a phenomenon where large language models (LLMs) or other AI systems generate false, misleading, or nonsensical information while presenting it as factual. This occurs when the AI model produces output that is not grounded in its training data or contradicts known facts.

AI hallucinations are responses generated by AI that contain inaccurate or fabricated information presented as truth. The causes are Insufficient or biased training data, limitations in model architecture, overfitting or incorrect pattern recognition, lack of context or understanding of underlying reality

1. Types of hallucinations: Factual inaccuracies, Sentence contradictions, Prompt contradictions, Irrelevant or random information generation. Hallucinations are estimated to occur in 15% to 20% of ChatGPT responses and other LLM responses.

Several strategies can be employed to detect hallucinations in RAG systems:

1. Automated fact comparison: Compare the AI's output to ground truth data or expected values
2. Specialized hallucination detectors: Use small, specialized models like p-faithful-v0 to classify outputs as faithful or hallucinations with high accuracy

p-faithful-v0 is a small, specialized language model designed for hallucination detection in Large Language Models (LLMs). Key points about p-faithful-v0 include:

• It can classify LLM outputs as faithful or hallucinations with accuracy similar to or better than much larger baseline models.
• It is a tiny fraction of the size of the LLMs backing baseline algorithms, offering significant inference cost savings.
• In out-of-domain testing, p-faithful-v0 performed comparably to or better than baseline models like RAGAS (using GPT-3.5) and Phoenix (using GPT-4).
• With modest fine-tuning on a small labeled training set, p-faithful-v0's performance improved from 70% to 93% accuracy on a challenging benchmark.
• It's designed for evaluating production RAG (Retrieval Augmented Generation) applications.
• The model aims to address the issue of LLMs generating unfaithful or hallucinatory responses, especially in real-world applications.

Other techniques:

1. Specific automated checks: Create evaluators that look for particular types of hallucinations common in your system
2. External knowledge verification: Cross-reference outputs with reliable external databases or knowledge graphs
3. Confidence scoring: Assign probability scores to outputs based on the model's confidence, flagging low-confidence outputs for review
4. Consistency checks: Generate multiple outputs for the same input and compare them for inconsistencies
5. Expert review: Have domain experts manually review a subset of outputs for hallucinations
6. End-user feedback: Implement mechanisms for users to flag potential hallucinations in the system's responses
7. LLM-as-a-judge: Use one model to evaluate another model's outputs, though this can be resource-intensive

It's important to note that no single method is perfect, and a combination of approaches may be necessary for effective hallucination detection in RAG systems

I studied and am reporting on two ways to mitigate the problem.

Option 1: Coaching an LLM to give the answers:

One LLM responded

I understand the task. You'd like me to compare a submitted answer to an expert answer for a given question, focusing solely on the factual content while ignoring stylistic differences. I should then categorize the comparison into one of five options (A through E) based on how the factual content relates between the two answers. The categories are:

A - The submission is a factual subset of the expert answer (omission)

B - The submission is a factual superset of the expert answer (potential hallucination)

C - The answers contain the same factual details

D - There's a factual disagreement between the answers

E - The answers differ in style only, not in facts. I'll wait for you to provide the specific question, expert answer, and submitted answer to analyze before making this comparison and categorization.

Please let me know when you have that data ready to share.

I then prompted it:

specific question: is reincarnation a fact, expert answer: yes it is, and submitted answer: reincarnation has been heavily discussed and does not agree with modern science

The LLM responded:

Based on the given information, I would select option (D) for this comparison: (D) There is a disagreement between the submitted answer and the expert answer. The expert answer states that reincarnation is a fact, while the submitted answer suggests that reincarnation does not agree with modern science. This represents a clear factual disagreement between the two answers. The submitted answer acknowledges that reincarnation has been heavily discussed, which is consistent with the information provided in the search results.

Option 2: LLM as a judge

This refers to the use of Large Language Models to evaluate content, responses, or performances, including the performance of other AI models. Key points about this approach include:

1. Applications: It can be used for evaluating AI model accuracy, automating grading in education, content moderation, and benchmarking.
2. Process: The LLM is given a specific judging task, a prompt with evaluation criteria, and the content to be judged. It then generates an output representing its judgment.
3. Types: There are three main types of LLM-as-a-judge evaluations: Single output scoring (without reference)Single output scoring (with reference)Pairwise comparison
4. Advantages: Scalability: Reduces the need for human involvement. Explainability: Provides reasoning behind judgments. Cost-effectiveness compared to human evaluation. Ability to understand complex text across various formats
5. Limitations and challenges: Potential biases (e.g., position bias, verbosity bias)Need for careful prompt engineering. Possibility of hallucinations or incorrect evaluations
6. Effectiveness: Research shows that state-of-the-art LLMs like GPT-4 can align with human judgment up to 85% of the time, which is higher than the agreement among humans (81%).
7. Improvement strategies: Chain-of-thought prompting for more accurate evaluations. Self-improving systems that incorporate human corrections as few-shot examples
8. Integration: LLM-as-a-judge can be integrated with other AI components to create more sophisticated gen AI architectures.

This approach offers a promising alternative to traditional human evaluations, especially for tasks requiring scalable and rapid assessment of language model outputs.

Lessons learned:

1. Coaching an LLM to Compare Answers

Pros:

• Accurate factual comparison
• Can detect stylistic differences

Cons:

• Requires expert answers for comparison
• Time-consuming and expensive
• Example: Your LLM accurately detected a factual disagreement between the expert answer and submitted answer.

2. LLM as a Judge

Pros:

• Scalable and cost-effective
• Provides explainability and reasoning
• Can evaluate complex text

Cons:

• Potential biases and hallucinations
• Requires careful prompt engineering

Effectiveness: State-of-the-art LLMs like GPT-4 can align with human judgment up to 85% of the time.

Recommendations

1. Hybrid Approach: Combine both methods to leverage their strengths. Use LLM-as-a-judge for initial evaluation and coaching an LLM for critical or disputed cases.
2. Fine-tune LLMs: Continuously fine-tune LLMs with diverse datasets and human corrections to improve accuracy and reduce hallucinations.
3. Chain-of-Thought Prompting: Implement chain-of-thought prompting to enhance LLM-as-a-judge evaluations.

Conclusion:

Hallucination can be mitigated by careful comparing the LLM answer with the Expert answer. This is however expensive and time consuming. LLM as a Judge might be an easier option or a hybrid option might be taken.

References:

1. https://www.ibm.com/topics/ai-hallucinations
2. https://builtin.com/artificial-intelligence/ai-hallucination

Image:

In Vedic Sanatana Hinduism, Shiva, the Demigod of destruction opens his third eye during Pralaya or Cosmic annihilation. This image is the artistic liberalism of that concept hinting at imminent societal destruction due to AI.