Understanding Hallucinations in Diffusion Models Through Mode Interpolation

Abstract

Diffusion models, known for their high-quality image generation, occasionally produce unexpected artifacts known as "hallucinations." These artifacts are images or parts of images that do not exist in the training data. This paper explores the cause of such hallucinations, focusing on "mode interpolation," where the model generates images by interpolating between different modes of the training data, resulting in unrealistic artifacts. Through systematic experiments, the study reveals that hallucinations stem from the model's inability to accurately capture the underlying data distribution, particularly in regions with discontinuous data modes. The research proposes a method to detect and mitigate these hallucinations, significantly improving the reliability of diffusion models.

Access the Paper Here.

Introduction

Diffusion models have emerged as the preferred generative models for tasks like image generation, inpainting, and super-resolution due to their ability to produce high-quality and diverse images. However, these models sometimes generate hallucinations—samples that fall outside the distribution of the training data. This phenomenon poses a significant problem as synthetic data generated by these models can influence subsequent models trained on such data, compounding errors over time. This paper aims to understand the root cause of hallucinations in diffusion models and proposes strategies to mitigate them.

The Phenomenon of Hallucinations

Hallucinations in diffusion models manifest as images containing artifacts or combinations of features not present in the training data. For example, a model trained on simple shapes might generate images with multiple instances of the same shape, a scenario absent from the original dataset. This suggests the model is interpolating between different modes of the data distribution, creating new, unrealistic samples.

Related Work

Diffusion models, introduced by the Paper progressively add noise to data and learn to reverse this process, effectively denoising the data. These models are related to score-based generative models and variational autoencoders (VAEs). Previous studies have explored various failure modes of diffusion models, such as training instabilities and unrealistic image generation. Recursive training, where generative models are trained on their own outputs, has been shown to lead to model collapse. This paper builds on these studies by focusing specifically on the hallucination phenomenon and its implications for recursive training.

Defining Hallucinations

A hallucination is formally defined as a sample generated by the model that lies entirely outside the support of the real data distribution. Mode interpolation is introduced to explain how these hallucinations occur. Mode interpolation happens when the model generates samples that lie between the modes of the data distribution, leading to artifacts that do not belong to any mode in the training data.

Experimental Setup

1D Gaussian Experiment

The first experiment uses a mixture of 1D Gaussians to illustrate mode interpolation. The dataset consists of three Gaussians with means at 1, 2, and 3, and a small standard deviation to ensure distinct modes. The diffusion model is trained on samples from this distribution, and the generated samples are analyzed. Results show that the model generates samples in regions between the Gaussians, confirming the mode interpolation hypothesis.

2D Gaussian Experiment

To further investigate, the study extends the experiment to 2D Gaussians arranged in a grid. The training set consists of 100,000 samples, and the model is trained to learn this distribution. The generated samples again show interpolation between the modes, confirming that mode interpolation occurs in higher dimensions as well.

SIMPLE SHAPES Dataset

The study uses a synthetic dataset of simple shapes (triangles, squares, pentagons) to simulate a more realistic scenario. The diffusion model trained on this dataset generates images with multiple instances of the same shape, highlighting how mode interpolation leads to unrealistic and unintended outputs in practical applications.

Detecting Hallucinations

The study proposes a method to detect hallucinations using the variance in the sampling trajectory of the diffusion model. Hallucinated samples show higher variance in their trajectory towards the final steps of the reverse diffusion process. Monitoring this variance allows for the identification and filtering of hallucinations during generation. Experiments demonstrate the method's effectiveness in removing over 95% of hallucinations while retaining 96% of valid samples, significantly improving the quality and reliability of the generated data.

Implications for Recursive Training

Recursive training, where a generative model is retrained on its own outputs, is susceptible to model collapse, exacerbated by hallucinations. The proposed detection method stabilizes recursive training by filtering out hallucinations, preventing the model from learning unrealistic artifacts. Experiments on datasets like MNIST and 2D Gaussians demonstrate the approach's effectiveness in mitigating model collapse.

Conclusion

Hallucinations in diffusion models arise from mode interpolation, where the model generates samples by interpolating between different modes of the data distribution. This paper provides a detailed analysis of this phenomenon and proposes a method to detect and mitigate hallucinations. The findings have significant implications for the stability and reliability of generative models, especially in recursive training scenarios. By addressing hallucinations, the study contributes to the development of more robust diffusion models, enhancing their utility in various applications.