Intro

Diffusion models are unsupervised probabilistic generative models inspired by non-equilibrium thermodynamics. They have 2 major processes:

1. Forward diffusion process: Adding noise to data until the result is no different to random noise.
2. Reverse diffusion process: Learning to reverse the diffusion process to reconstruct the original data from the noise obtained from forward diffusion process.

Forward diffusion process

Given an image, which we call $x_0$, the forward diffusion process is in which we sample a series of Gaussian noise and add them to the image in $T$ steps, producing a sequence of noisy samples $x_1, x_2, …, x_T$. Eventually when $T \rightarrow \infty$, $x_T$ is equivalent to an isotropic Gaussian distribution, which will be proved later.

Let’s assume each step can be pictured like this:

  
flowchart LR
	input(("x(t-1)")) --> processor("q(x(t)|x(t-1))")
	processor --> output(("x(t)"))

  

Each step is controlled by a variance schedule $\{\beta_t \in (0, 1)\}^T_{t=1}$.

In this equation, we can tell $\sqrt{1-\beta_t}x_{t-1}$ is the mean and $\beta_t\mathbf{I}$ is the variance. While the mean is smaller than 1, each step will shrink the value for each pixel in the image, and then add a noise with variance equals $\beta_t$.

At the beginning, small noise can cause considerable impact, so we can make the $\beta_t$ small to keep the gap between $x_t$ and $x_{t-1}$ closer. And with time pass by, we will need bigger $\beta_t$ to ensure changes could be tell. We can set $\beta_1 = 10^{-4}$ and $\beta_T = 0.02$.

It’s not difficult to find that $x_t$ only depends on $x_{t-1}$, therefore the forward diffusion process is Markov process.

Re-parameterization

Before diving into the calculation, it’s better to introduce re-parameterization, which will be used during the calculation.

Let’s assume a random value $x \sim \mathcal{N}(\mu, \sigma^2 \mathbf{I})$, let $\epsilon \sim \mathcal{N}(0, \mathbf{I})$, then $x$ can be re-parameterized as $z = \mu + \sigma \epsilon$.

Re-parameterization can transfer the randomicity from $x$ to $\epsilon$, which can not only simplify the calculation, but make the reverse diffusion process differentiable.

Proof

Let $\alpha_t = 1 - \beta_t$, $\bar{\alpha}_t = \prod^t_{i=1}\alpha_i$.

Re-parameterize the equation (1) we can get:

Considering $x_1 = \sqrt{\alpha_1} x_0 + \sqrt{1-\alpha_1} \epsilon$, we get:

when $T \rightarrow \infty$, we get:

Reverse diffusion Process

If we can reverse the forward diffusion process, we will be able to recreate the origin image from the Gaussian noise. But we cannot easily estimate $q(x_{t-1}|x_{t})$ because it needs to use the entire dataset. Therefore, we need to learn a model $p_{\theta}$ to approximate these conditional probabilities in order to run the reverse diffusion process.

  
flowchart LR
	subgraph Theory
		input1(("x(t)")) --> processor1("q(x(t-1)|x(t))")
		processor1 --> output1(("x(t-1)"))
	end
	subgraph Practice
		input2(("x(t)")) --> processor2("p(x(t-1)|x(t))")
		processor2 --> output2(("x(t-1)"))
	end
	Theory-->Practice

  

Now the problem is how to learn the model $p_{\theta}$.

It is noteworthy that the reverse conditional probability is tractable when conditioned on $x_0$:

Using Bayes’ rule, we have:

From the forward diffusion process, we can get:

Combine the result above and because we only concern $x_{t-1}$ , we have:

where $C(x_t,x_0)$ is some function not involving $x_{t-1}$ and details are omitted. Following the standard Gaussian density function, the mean and variance can be parameterized as follows:

From (2) we can represent $x_0=\frac{x_t-\sqrt{1-\bar{\alpha}_t}\epsilon}{\bar{\alpha}_t}$, and plug it into the above equation and obtain:

References

1. What are Diffusion Models? | Lil’Log (lilianweng.github.io)
2. 扩散模型/Diffusion Model原理讲解