Fine-Tuning SLMs for Enterprise-Grade Evaluation & Observability

Session slides: https://pbase.ai/46Z4cXQ

YouTube: https://youtube.com/watch?v=E4SWbTAaIog

At the recent LLMOps Micro-Summit, Atindriyo Sanyal, Co-founder & CTO of Galileo, delved into cutting-edge techniques for tackling hallucinations in large language models (LLMs). His focus was on new methods for fine-tuning small language models (SLMs) to enhance the observability and evaluation of these models. The talk offered a comprehensive look at the evolution of evaluating hallucinations in language models and introduced novel approaches like ChainPoll and Luna that are poised to set new standards in the AI landscape.

Understanding Hallucinations in LLMs

Atin began his talk by addressing a term that has become somewhat of a cliché in the world of LLMs: hallucinations. He referenced a famous quote by Andrej Karpathy, who once said that "everything an LLM says is a hallucination." This idea has sparked much debate about whether these “dreams” generated by LLMs are a feature or a bug. Hallucinations can be both—a manifestation of the creativity and complexity of these models, but also a risk factor when it comes to reliability and accuracy.

As we move deeper into the era of non-deterministic software, where AI systems are embedded in enterprise applications, the challenges surrounding hallucinations are becoming more pressing. Recent industry insights, such as the McKinsey State of AI Report 2024, highlight that inaccuracies in model generation and intellectual property infringement have seen a marked increase in importance among practitioners.

Traditional Methods of Detecting Hallucinations

Atin outlined the three most common techniques currently used to detect hallucinations in LLMs:

1. N-gram Matching: This technique uses methods like BLEU and ROUGE scores to compare the generated text with reference completions. While useful for certain applications like machine translation or summarization, these methods fall short in real-world scenarios where the models’ outputs are more nuanced and creative. They are overly reliant on precise token matches and require ground truth datasets, which are not always available.
2. Asking GPT or LLMs as Judges: This method involves using a more advanced LLM to evaluate the outputs of another LLM. While it can provide some insights, this approach is black-boxed, lacks explainability, and can be prohibitively expensive, especially when scaling up to millions of queries per day.
3. Human Evaluations: Despite technological advancements, human evaluations are still widely used. However, they are slow, costly, and often biased due to inconsistent criteria among evaluators.

The Need for New Evaluation Metrics

Given the limitations of these traditional methods, Atin emphasized the need for a new category of evaluation metrics that are:

• Highly Accurate: Ensuring the outputs are genuinely reflective of the desired outcomes.
• Scalable: Applicable across diverse and real-world tasks without requiring extensive ground truth datasets.
• Cost-Effective: Minimizing the financial burden on organizations, especially as they scale up LLM deployments.
• Low Latency: Enabling real-time evaluations to support production applications.

Introducing ChainPoll: An Innovative Approach

To address these challenges, Atin introduced the first of two new techniques developed at Galileo—ChainPoll. ChainPoll builds upon the idea of using another LLM for hallucination detection but adds several layers of sophistication:

• Chain of Thought Prompting: This method involves guiding the LLM through a step-by-step reasoning process to detect hallucinations with higher efficacy.
• Polling or Ensembling: Instead of a single binary classification, multiple calls are made to the LLM, and a majority voting mechanism is used to enhance accuracy. This ensemble approach significantly improves the reliability of hallucination detection.

ChainPoll is designed as a framework rather than a standalone LLM, allowing it to be easily integrated with various models and techniques. One notable advantage of ChainPoll is its flexibility and adaptability. For example, one of Galileo's clients was able to replace the default ChainPoll model (GPT-3.5) with GPT-4 Mini, resulting in a 60-70% reduction in costs without sacrificing accuracy.

Introducing Luna: The Power of Small Language Models (SLMs)

Moving beyond ChainPoll, Atin introduced Luna, a small language model (SLM) fine-tuned specifically to detect hallucinations in Retrieval-Augmented Generation (RAG) use cases. Luna is a 400-million-parameter model, significantly smaller than traditional LLMs, but it has been meticulously trained on high-quality hallucination data collected over several years.

Luna utilizes a unique approach known as Natural Language Inference (NLI) to determine if the generated output is consistent with the provided context. This makes it particularly effective for RAG use cases where it is essential to ensure that the AI-generated response aligns closely with the input context.

Key Innovations in Luna

1. Novel Windowing Approach: Luna divides RAG contexts and generations into overlapping segments. This segmented approach allows for sentence-level hallucination detection, providing enhanced explainability and precision in outputs.
2. Multitask Training: Luna is designed to provide multiple evaluation metrics—such as adherence, utilization, and relevance—in a single inference call. This approach ensures a comprehensive evaluation of the AI system, considering various dimensions beyond mere accuracy.
3. Synthetic Data and Augmentation: Luna leverages synthetic data generation and augmentation techniques to improve robustness and domain coverage. Inspired by methods used in computer vision, this approach enables Luna to understand nuanced language variations that are typically handled better by larger models.
4. Cost and Latency Advantages: Luna operates at a fraction of the cost of traditional LLMs like GPT-3.5 while offering ultra-low latency, making it suitable for real-time applications. As Atin highlighted, this is a significant achievement given the increasing need for efficient and scalable AI solutions.

Real-World Impact and Use Cases

One of the most compelling aspects of Luna is its adaptability. As Atin mentioned in response to an audience question from Spark, a solution architect:

“What we’ve seen is a 78% baseline accuracy for Luna, which is pretty state-of-the-art. But by fine-tuning it on about 150 samples of production data, one of our customers was able to increase that accuracy to 98%. The last mile is critical, and fine-tuning is key to getting there.”

This adaptability makes Luna particularly valuable for enterprises looking to implement real-time AI solutions with high reliability. From enhancing customer service chatbots to ensuring the factual correctness of generated content, Luna provides a robust foundation for evaluating and mitigating hallucinations.

Consider the Following Strategic Actions:

1. Adopt a Multi-Layered Evaluation Approach: Instead of relying on a single technique, consider integrating multiple methods like ChainPoll and Luna to achieve higher accuracy in detecting hallucinations.
2. Leverage Small Language Models (SLMs) for Cost-Effective Solutions: Invest in fine-tuning smaller models like Luna, which offer a balance between accuracy, cost, and latency—essential for scaling AI in production environments.
3. Implement Real-Time Guardrails: As the need for real-time AI evaluation grows, consider deploying solutions like Galileo's Luna Suite that provide immediate feedback and corrective actions.
4. Customize and Fine-Tune for Last-Mile Accuracy: Recognize that fine-tuning is crucial for adapting models to specific business needs and achieving optimal performance.
5. Explore Synthetic Data Generation: Use synthetic data to fill gaps in real-world datasets, especially when training smaller models. This can improve model robustness and generalizability across diverse scenarios.

By focusing on these strategic actions, CTOs and executives can better navigate the complexities of deploying and scaling generative AI technologies in an enterprise context.

These insights and innovations shared by Atin Sanyal at the LLMOps Micro-Summit provide a roadmap for leveraging the next generation of AI tools to achieve enterprise-grade evaluation and observability. As the AI landscape evolves, adopting these advanced techniques for fine-tuning and monitoring small language models (SLMs) will be critical for staying ahead in a competitive market. However, building high-performing models also depends on the quality and diversity of training data. In our next article, we will explore how synthetic data can be used to build better models faster, offering a powerful approach to further enhance AI performance and adaptability. Stay tuned for more on "Building Better Models Faster with Synthetic Data."