What are Diffusion Models?

Diffusion models is one of the hottest topics now. This short post is just a reminder what is this and how they emerged and had been developed.

Forward diffusion process

Given a data point sampled from a real data distribution, let us define a forward diffusion process in which we add a small amount of Gaussian noise to the sample in steps, producing a sequence of noisy samples . The step sizes are controlled by a variance schedule?'Beta'.

Reverse diffusion process

If we can reverse the above process and sample from generated data.

We will be able to recreate the true sample from a Gaussian noise input.

Research and development effort of DF include following topics:

1. Parameterization of for training loss.
2. Parameterization of variance schedule 'Beta'? (The forward variances are set to be a sequence of linearly increasing constants from 1e-4 to 0.02).
3. Parameterization of reverse process variance.
4. Conditioned Generation (While training generative models on images with conditioning information such as ImageNet dataset, it is common to generate samples conditioned on class labels or a piece of descriptive text).
5. Speed up Diffusion Models is a broad topic which includes: a) Fewer Sampling Steps (One simple way is to run a strided sampling schedule by taking the sampling update every S steps to reduce the process from T to S steps); b) Distillation (Distilling trained deterministic samplers into new models. In every progressive distillation iteration, we can half the sampling steps.).
6. Latent Variable Space (Latent diffusion model runs the diffusion process in the latent space instead of pixel space, making training cost lower and inference speed faster).?
7. Model Architectures.

There are two common backbone architecture choices for diffusion models: U-Net and Transformer.

Quick Summary

Pros: Tractability and flexibility are two conflicting objectives in generative modeling. Tractable models can be analytically evaluated and cheaply fit data (e.g. via a Gaussian or Laplace), but they cannot easily describe the structure in rich datasets. Flexible models can fit arbitrary structures in data, but evaluating, training, or sampling from these models is usually expensive. Diffusion models are both analytically tractable and flexible

Cons: Diffusion models rely on a long Markov chain of diffusion steps to generate samples, so it can be quite expensive in terms of time and computation. New methods have been proposed to make the process much faster, but the sampling is still slower than GAN.

Source: Weng, Lilian. (Jul 2021). What are diffusion models? Lil’Log. https://lilianweng.github.io/posts/2021-07-11-diffusion-models/.