7.4 Generative AI and diffusion🔗⧉

A restoration algorithm can return one estimate, or generate several plausible explanations of the observation. Generative models make the second option practical. Their usefulness depends on what they have learned about images and how faithfully the reconstruction obeys the measurement.

7.4.1 The framing: generation is learning and sampling a prior $p(x)$🔗⧉

A generative model provides a procedure for drawing images from a model distribution $p_\theta(x)$. A Gaussian prior can already be sampled; so can the hand-built dead-leaves model in Chapter 7.5, Image priors. Learned generators aim to model richer image structure. Evaluating a penalty, applying a denoiser, and sampling a distribution are different operations; possessing one does not automatically provide the others (Figure 7.4.1⛶ fullscreen).

The model may generate unconditionally, $x\sim p_\theta(x)$, or use a condition such as text, a sketch, or a measurement. For restoration, the desired distribution is the posterior $p(x\mid y)\propto p(y\mid x)p(x)$. A conditional generator approximates that posterior only if its training or sampling procedure accounts for the measurement model.

A thumbnail or masked image can correspond to several plausible originals. An ideal squared-error regressor returns their conditional mean, which can blur uncertain detail. A conditional sampler instead draws alternatives from its model. Sharpness, fidelity, and diversity depend on that model; sampling does not guarantee a correct or sharp reconstruction.

Efros–Leung texture synthesis Efros & Leung 1999 samples pixels whose neighborhoods resemble already synthesized surroundings in an example texture. It estimates local appearance statistics by lookup. Learned generative models extend the range of structures that can be represented, rather than introducing sampling itself.

Consider filling a hole in a brick wall. Several alignments or textures may fit the visible boundary, and averaging them can produce an implausible smooth patch. Sampling attempts to choose a coherent completion instead. Running the method again may give another completion with similar plausibility. This is useful when an image is being edited, but it does not establish which bricks were behind the removed object. Restoration adds the further requirement that the chosen completion remain consistent with the measurements that are available.

7.4.2 Diffusion: generation as iterated denoising🔗⧉

Diffusion generates an image by reversing a gradual corruption process. Training starts with photographs and adds noise; sampling starts with noise and applies a learned reverse process (Figure 7.4.2⛶ fullscreen).

The forward chain gradually attenuates the image and adds independent Gaussian noise. Define $\alpha_t=1-\beta_t$ for noise variances $0<\beta_t<1$, and $\bar\alpha_t=\prod_{s=1}^t\alpha_s$, with $\bar\alpha_0=1$. A sample at any level can be generated directly:

The clean contribution is scaled by $\sqrt{\bar\alpha_t}$. Choosing $\bar\alpha_T$ close to zero makes the endpoint approximately standard Gaussian; the approximation is not exact at a finite nonzero signal level. The name variance-preserving refers to preserving unit marginal variance when the data have unit variance. The reverse model learns transitions from $x_t$ toward $x_{t-1}$.

One common parameterization predicts the added noise with a network $\epsilon_\theta(x_t,t)$. The DDPM simplified training objective averages over training images, Gaussian noise, and sampled levels Ho et al. 2020:

This is squared-error regression. Converting its prediction to a clean-image estimate requires both subtraction and rescaling:

A U-Net or transformer can implement the predictor; the noise level is an essential input.

The key identity ties diffusion back to the rest of the book. For a Gaussian-noised image under the variance-preserving forward process, the best possible denoiser, the one returning the posterior-mean estimate of the clean image, is given by Tweedie's formula:

The score $\nabla_{x_t}\log p_t(x_t)$ describes the density of images at noise level $t$. For an exact minimum-MSE predictor it equals $-\mathbb E[\epsilon\mid x_t]/\sqrt{1-\bar\alpha_t}$. Thus $(1-\bar\alpha_t)$ times the score points from $x_t$ to the rescaled posterior mean $\sqrt{\bar\alpha_t}\,\hat x_0$, not generally to $\hat x_0$ itself. The trained predictor approximates this identity (Figure 7.4.3⛶ fullscreen; Song et al. 2021).

Spelled out as a loop, sampling starts from pure static and walks the schedule down to a clean image:

x ← sample N(0, I)
for t = T, ..., 1:
    eps ← epsilon_theta(x, t)
    mu ← (x - beta[t] * eps / sqrt(1 - alpha_bar[t])) / sqrt(alpha[t])
    variance ← beta[t] * (1 - alpha_bar[t-1]) / (1 - alpha_bar[t])
    x ← mu + sqrt(variance) * sample N(0, I) if t > 1 else mu
return x

This is ancestral DDPM sampling with the posterior-variance choice $\tilde\beta_t=\beta_t(1-\bar\alpha_{t-1})/(1-\bar\alpha_t)$. Other variance choices and samplers use different updates. Skipping levels requires the corresponding sampler coefficients; simply omitting iterations from this loop changes the process.

Images are too high-dimensional to watch this happen all at once, so it helps to run the exact same loop on a distribution simple enough to see directly: a 2-D point cloud.

Diffusion and plug-and-play restoration both use denoisers, but their iterations have different purposes. A plug-and-play solver seeks a reconstruction or fixed point. A diffusion sampler transports a random initial draw through a specified sequence of distributions. Adding a denoiser to a solver does not turn it into a posterior sampler (Figure 7.4.10⛶ fullscreen).

Latent diffusion Rombach et al. 2022 reduces spatial cost by encoding an image as a smaller tensor $z=\mathcal E(x)$, generating in that latent space, and decoding with $\mathcal D$. Compression can discard visible texture or alter fine geometry. The autoencoder therefore sets part of the reconstruction limit before diffusion begins (Figure 7.4.5⛶ fullscreen).

7.4.3 Conditioning: text, images, and control🔗⧉

So far the model samples from $p(x)$, any plausible image. The capability everyone actually wants is to steer it: sample from $p(x \mid c)$ for a condition $c$ of your choosing.

A text encoder turns the prompt into features that condition the denoiser, for example through cross-attention. CLIP Radford et al. 2021 (CLIP) is one option. The resulting conditional model learns associations between text and images; a prompt is not a precise geometric or physical constraint.

Both images and text can be represented as discrete tokens, and image generators can be autoregressive. Diffusion instead updates a continuous image or latent tensor jointly. Text conditioning does not require a shared CLIP embedding space: a suitable text encoder can supply features that the image model learns to use.

How hard the model obeys the prompt is itself a knob, and the standard one is classifier-free guidance (CFG) (Ho & Salimans 2022). Train the model both with and without the condition. During training, drop $c$ some fraction of the time and replace it with a null token $\varnothing$, so a single network can predict noise both ways. At sampling time, run it twice, conditioned and unconditioned, and extrapolate along the difference:

Here $w=0$ uses the unconditional prediction, $w=1$ uses the conditional prediction, and $w>1$ extrapolates beyond it. Larger weights can increase prompt adherence while reducing diversity or causing saturation and artifacts. Guided sampling generally changes the distribution; it is not exact sampling of the original conditional model.

Text is not the only useful condition. Sometimes you want to fix the structure of the output, such as a pose, an outline, or a depth layout, while letting the prompt fill in content. ControlNet (Zhang et al. 2023) adds a trainable side branch that injects a spatial hint (edges, human pose, a depth map, a segmentation) into the frozen base model, to encourage agreement with the hint and prompt without retraining the whole network (Figure 7.4.6⛶ fullscreen). And InstructPix2Pix (Brooks et al. 2023) conditions on a pair (an input image, a text instruction) and edits in place: "make it winter," "turn the car red." This is the diffusion descendant of the paired and unpaired image-to-image translation (Pix2Pix, CycleGAN) we met in Chapter 7.3, Deep learning, now driven by natural-language instructions.

An autoregressive image model needs a chosen order of tokens and predicts each one conditional on the preceding tokens. A diffusion model updates a noisy spatial array at each time level, so the order is over noise levels rather than over pixel locations. Both choices can represent image distributions. Their computational dependencies differ, which helps explain differences in sampling latency and conditioning interfaces without making either representation mandatory for the modality.

7.4.4 Posterior sampling: generative priors for inverse problems🔗⧉

For restoration, the condition is a measurement with a known degradation and noise model.

For $y=Ax_0+n$, the target is $p(x_0\mid y)\propto p(y\mid x_0)p(x_0)$. At an intermediate diffusion level the relevant identity is

The second term involves uncertainty about the clean image: $p_t(y\mid x_t)=\int p(y\mid x_0)p(x_0\mid x_t)\,dx_0$. It is generally not the likelihood obtained by applying $A$ directly to the noisy $x_t$. Practical guidance methods approximate this term or construct specialized conditional transitions. Alternating denoising and a data-fit correction can be useful, but does not by itself guarantee posterior samples (Figure 7.4.9⛶ fullscreen).

Detail unconstrained by the measurement comes from the prior. Different samples expose some of the model’s uncertainty, but their spread is calibrated only if the prior, likelihood, and sampler are adequate. Check residuals against the assumed noise, and distinguish observed structure from generated alternatives.

Figure 7.4.10⛶ fullscreen compares ways of supplying image knowledge to an algorithm. It should be read as a collection of interfaces, not a ranking that guarantees better reconstruction as model complexity increases.

7.4.5 Other generative families, in brief🔗⧉

Generative families differ in their training objectives, representations, and sampling costs.

Generative adversarial networks (GANs; Goodfellow et al. 2014) can generate in one forward pass. Adversarial training can be unstable or miss modes. Diffusion uses a regression objective and typically several network evaluations; this does not guarantee complete coverage of the image distribution. Distillation and accelerated samplers further change the speed comparison.

Variational autoencoders (VAEs; Kingma & Welling 2014) learn an encoder, a latent prior, and a probabilistic decoder. Sample sharpness depends on the likelihood and training objective. Latent diffusion can reuse an autoencoder trained with KL regularization, but neither all autoencoders nor all latent diffusion models are identical to a basic VAE.

Autoregressive / transformer image models generate tokens (raw pixels, or entries from a learned codebook) one at a time, $p(x) = \prod_i p(x_i \mid x_{<i})$, the very same next-token recipe as a language model (PixelRNN/CNN van den Oord et al. 2016 (PixelRNN), image transformers, vector-quantization based models). They are slow to sample but braid images and text into one token stream, which is why they are central to multimodal systems.

Flow matching trains a velocity field that transports a simple initial distribution toward the data distribution; sampling integrates an ordinary differential equation. Its prediction target and time convention differ from the DDPM noise-prediction loop above. The checkpoint and scheduler must agree on that interface (Lipman et al., Flow Matching). Deterministic diffusion sampling is also developed in DDIM.

7.4.6 Caveats and ethics🔗⧉

The same property that makes a generative prior strong makes it untrustworthy as evidence, and the tension is structural, not a bug to be fixed.

Hallucination is built in. A sampleable prior must invent detail. That is what lets it produce a sharp answer to a one-to-many problem. Generated and super-resolved detail is therefore plausible, not real, and must never be treated as a measurement in forensic, medical, or legal settings. This is L7.8show's reconstruction-versus-hallucination split with the dial turned to maximum, and it carries forward to Chapter 40.1, Human factors and the art of photography.

Training data and objectives affect which subjects, appearances, and compositions the model favors. The model can combine features into novel images, while still reproducing dataset biases.

Deepfakes, provenance, and consent are unsettled. Realistic synthesis enables convincing deepfakes and misinformation; the countermeasures are partial: watermarking, provenance standards like the C2PA (Coalition for Content Provenance and Authenticity) effort, learned detectors (the forensics angle, cross-referencing the face-landmark consistency checks of Chapter 7.3, Deep learning). And training on billions of scraped images raises copyright, attribution, and consent questions that remain open.

The costs concentrate. Training and serving large diffusion models is compute- and energy-intensive, so the capability pools with whoever can pay. This is the same caveat raised by the Bitter Lesson in Chapter 7.2, Machine learning. Flag it; don't cheerlead.

For practical use, record the checkpoint, preprocessing, prediction parameterization, scheduler, step count, guidance weight, and random seed. A sampler expecting noise predictions cannot silently accept clean-image or velocity predictions. Assess speed together with fidelity and diversity.

The web demonstration should distinguish a schematic trajectory from actual model inference. A precomputed or simulated sequence illustrates the process but does not demonstrate prompt conditioning or reconstruction quality.