Guide to Metrics and Thresholds for Evaluating RAG and LLM Models

Introduction

This guide provides a comprehensive overview of various metrics used for evaluating Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs). The accompanying code calculates and visualizes these metrics, offering insights into the performance, diversity, relevance, and other critical aspects of the models.

Accompanying code: https://github.com/KevinAmrelle/LLM_RAG/blob/main/Rag_Eval_v2.ipynb

Metrics Overview

Basic Performance Metrics

·?????? Accuracy: Measures the proportion of correct predictions among the total number of cases. Best for evaluating classification models where correct labeling is crucial.

·?????? Precision: Evaluates the proportion of true positive predictions among all positive predictions. Important for tasks where false positives are costly.

·?????? Recall: Assesses the model's ability to identify all actual positives. Useful in scenarios where missing true positives is critical.

·?????? F1 Score: Balances precision and recall, making it suitable for datasets with uneven class distributions.

Advanced Composite Metrics

·?????? F2 Score: Emphasizes recall over precision. Ideal for applications where capturing all positives is more critical than precision.

·?????? F0.5 Score: Prioritizes precision over recall. Suitable for tasks where false positives need to be minimized.

·?????? BLEU (Bilingual Evaluation Understudy): Focuses on the similarity between machine-generated and human reference text, commonly used in translation tasks.

·?????? ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Measures the overlap of n-grams between the generated text and reference text, used for summarization tasks.

·?????? METEOR (Metric for Evaluation of Translation with Explicit ORdering): Considers synonymy and paraphrasing, aligning closely with human judgment in translation tasks.

·?????? BERTScore: Uses contextual embeddings from models like BERT to assess semantic similarity. Suitable for evaluating text generation and understanding.

?

Probability and Uncertainty Metrics

·?????? Cross-Entropy: Measures the dissimilarity between the predicted and actual probability distributions. Useful for evaluating probabilistic models.

·?????? Per-token Perplexity: Provides perplexity calculations at the token level, indicating how well a probability model predicts a sample.

Diversity and Novelty Metrics

·?????? Distinct-n: Quantifies the diversity of n-grams in the generated text. Higher values indicate more diverse text.

·?????? Self-BLEU: Assesses how repetitive or unique the text is relative to itself. Lower values indicate higher diversity.

Ranking and Retrieval Metrics

·?????? Mean Reciprocal Rank (MRR): Measures the average reciprocal ranks of results. Used in information retrieval and question-answering systems.

·?????? Hit Rate at K (Hit@K): Checks if the correct answer is within the top K results. Relevant for ranking systems.

·?????? Area Under the Curve (AUC): Measures the model's ability to distinguish between classes in binary classification tasks.

Semantic and Contextual Evaluation Metrics

·?????? Semantic Similarity: Evaluates how semantically similar phrases or texts are to each other. Useful for tasks requiring understanding of meaning.

·?????? Jaccard Index: Measures similarity and diversity between sample sets. Commonly used in clustering and similarity tasks.

RAG-specific Metrics

·?????? Toxicity: Assesses the presence of toxic content in generated text. Important for ensuring safe and appropriate model outputs.

·?????? Hallucination: Measures the proportion of generated content not present in the reference text. Critical for maintaining factual accuracy.

·?????? Relevance: Evaluates the relevance of the generated text to the reference text. Essential for generating contextually appropriate responses.

?

?

?

Threshold Logic for Metrics

In the evaluation of machine learning models, especially for RAG and LLM models, setting target thresholds for metrics helps to define what constitutes acceptable or excellent performance. These thresholds are benchmarks that provide guidance on the expected performance levels. Here’s a detailed explanation of the threshold logic for each metric:

?

Basic Performance Metrics

·?????? Accuracy (0.9)

o?? Threshold Logic: An accuracy of 90% or higher is generally considered good for classification tasks, indicating that the model correctly predicts the class for 90% of the cases.

o?? Application: Classification models where high correctness is crucial.

·?????? Precision (0.8)

o?? Threshold Logic: A precision of 80% indicates that 80% of the positive predictions made by the model are correct. This is particularly important in tasks where false positives need to be minimized.

o?? Application: Models where false positives are costly, such as medical diagnoses or spam detection.

·?????? Recall (0.8)

o?? Threshold Logic: A recall of 80% means the model successfully identifies 80% of the actual positives. This threshold is crucial for tasks where missing true positives is critical.

o?? Application: Use cases like fraud detection or disease screening where missing a positive case can have severe consequences.

·?????? F1 Score (0.85)

o?? Threshold Logic: An F1 score of 85% or higher indicates a good balance between precision and recall, suitable for datasets with imbalanced classes.

o?? Application: General classification tasks, especially with imbalanced data.

Advanced Composite Metrics

·?????? F2 Score (0.8)

o?? Threshold Logic: Emphasizing recall over precision with an 80% threshold ensures the model captures the majority of positive cases.

o?? Application: Scenarios where recall is more critical than precision, such as safety-critical applications.

·?????? F0.5 Score (0.8)

?

o?? Threshold Logic: Prioritizing precision with an 80% threshold reduces the number of false positives.

o?? Application: Applications like email filtering, where false positives (incorrectly labeled spam) need to be minimized.

·?????? BLEU (0.5)

o?? Threshold Logic: A BLEU score of 0.5 or higher indicates a moderate to high degree of similarity between the generated text and human reference text.

o?? Application: Translation and text generation tasks.

·?????? ROUGE (0.5)

o?? Threshold Logic: A ROUGE score of 0.5 indicates that there is a significant overlap between the generated summary and the reference summary.

o?? Application: Summarization tasks.

·?????? METEOR (0.5)

o?? Threshold Logic: A METEOR score of 0.5 or higher suggests that the generated text aligns well with human judgment, considering synonyms and paraphrases.

o?? Application: Translation and paraphrasing tasks.

·?????? BERTScore (0.85)

o?? Threshold Logic: A BERTScore of 0.85 indicates high semantic similarity between the generated text and the reference text.

o?? Application: Evaluating semantic similarity in text generation and understanding tasks.

Probability and Uncertainty Metrics

·?????? Cross-Entropy (0.3)

o?? Threshold Logic: A cross-entropy loss of 0.3 or lower indicates that the predicted probability distributions are close to the true distributions.

o?? Application: Probabilistic models and classification tasks.

·?????? Per-token Perplexity (20)

o?? Threshold Logic: A per-token perplexity of 20 or lower suggests that the model predicts the next token with a high degree of confidence.

o?? Application: Language modeling tasks.

?

?

?

Diversity and Novelty Metrics

·?????? Distinct-1 and Distinct-2 (0.5)

o?? Threshold Logic: A distinct-n score of 0.5 or higher indicates a good level of diversity in the generated text.

o?? Application: Text generation tasks where diversity is important.

·?????? Self-BLEU (0.3)

o?? Threshold Logic: A self-BLEU score of 0.3 or lower suggests that the generated text is not overly repetitive.

o?? Application: Evaluating the novelty of generated text.

Ranking and Retrieval Metrics

·?????? Mean Reciprocal Rank (MRR) (0.8)

o?? Threshold Logic: An MRR of 0.8 indicates that the correct answer appears high in the ranking order.

o?? Application: Information retrieval and question-answering systems.

·?????? Hit Rate at K (Hit@K) (0.8)

o?? Threshold Logic: A Hit@K of 0.8 means that the correct answer is found within the top K results 80% of the time.

o?? Application: Ranking systems.

·?????? Area Under the Curve (AUC) (0.85)

o?? Threshold Logic: An AUC of 0.85 or higher indicates good discriminative ability between the classes.

o?? Application: Binary classification tasks.

Semantic and Contextual Evaluation Metrics

·?????? Semantic Similarity (0.8)

o?? Threshold Logic: A semantic similarity score of 0.8 suggests high semantic congruence between the reference and generated text.

o?? Application: Text understanding and generation tasks.

·?????? Jaccard Index (0.8)

o?? Threshold Logic: A Jaccard Index of 0.8 indicates a high degree of overlap between the sets.

o?? Application: Clustering and similarity tasks.

?

?

?

RAG-specific Metrics

·?????? Toxicity (0.2)

o?? Threshold Logic: A toxicity score of 0.2 or lower ensures the generated text contains minimal toxic content.

o?? Application: Ensuring safe and appropriate content generation.

·?????? Hallucination (0.1)

o?? Threshold Logic: A hallucination score of 0.1 or lower suggests minimal generation of false or fabricated content.

o?? Application: Maintaining factual accuracy in generated content.

·?????? Relevance (0.8)

o?? Threshold Logic: A relevance score of 0.8 or higher indicates that the generated text is highly relevant to the reference text.

o?? Application: Generating contextually appropriate responses.