BookNotesSlidesP-sets Syllabusdraft · v0.2.49.1
◀▲⌂▶
💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.

13.9 Generative denoising with diffusion models⧉

In a very noisy photograph of a brick wall, several positions of a mortar line may be compatible with the measurement. Averaging those possibilities produces a softened edge. Choosing one produces a sharper edge whose position may be wrong. Generative denoising makes this choice explicit by producing plausible reconstructions conditioned on the noisy image.

The distinction matters where the observation leaves substantial uncertainty. It is not true that all posterior means look blurred or all posterior samples look realistic. Both depend on the scene, measurement, and learned distribution. A model that excludes the true structure can produce confident, consistent errors.

13.9.1 Why the posterior mean can blur detail⧉

For a fixed observation $y$, let $\mu(y)=\mathbb E[x\mid y]$. Any deterministic estimate $a$ satisfies

$$\mathbb E[\lVert x-a\rVert^2\mid y] =\mathbb E[\lVert x-\mu(y)\rVert^2\mid y]+\lVert a-\mu(y)\rVert^2.$$

The cross term vanishes because $\mathbb E[x-\mu(y)\mid y]=0$. Thus the posterior mean minimizes expected squared error. If likely images disagree on fine texture or edge position, their mean smooths those differences. This is one source of softness in squared-error denoisers, alongside limited model capacity and training mismatch.

For a scalar example, suppose a detail is equally likely to have value 0 or 1 after observing $y$. The mean is 0.5 and its expected squared error is 0.25. An independent posterior draw returns 0 or 1 and has expected error 0.5 against the unknown truth. It nevertheless matches the truth exactly on half the draws. A sample can beat the mean on a particular case; the optimality is an expectation, not an ordering of every result.

The perception-distortion tradeoff formalizes the tension between expected reconstruction error and agreement between output and natural-image distributions (Blau & Michaeli 2018; Chapter 14.6, The perception-distortion tradeoff and perceptual super-resolution). Lowering squared error need not make uncertain texture look more photographic. Nor does matching an image distribution certify the accuracy of a particular reconstruction.

fig-posterior-samples-vs-mean
Figure 13.9.1. Interactive. One noisy input, several posterior samples, and their mean, all on screen. The samples are exact draws from a simple model, a Gaussian prior in the wavelet basis with per-band variances fitted to the noisy input, so they are plug-in draws: exact for that fitted model, with the fit's own uncertainty not represented. At low noise the samples are nearly identical and the mean looks like any of them. Raise the noise and the samples visibly disagree, in texture first and then in structure, while the mean smooths in exactly the places they disagree. A per-pixel view of the sample variance is the map of what the measurement did not determine, and it is a more informative output than any single result. The coarse coefficients are held fixed across the draws, and these are samples from that fitted wavelet model rather than diffusion outputs or calibrated uncertainty. On a given reference, one sample can score better than the mean. Interactive: set the noise level, draw more samples, switch between samples, their mean and their variance, and read the score of each sample against the truth. The exact posterior mean minimizes expected squared error, so the mean of the draws usually scores better than any single draw, but not necessarily better than every one.

13.9.2 Sampling instead of averaging⧉

Bayes' rule describes the desired conditional distribution:

$$p(x\mid y)\propto p(y\mid x)p(x).$$

For additive white Gaussian measurement noise of standard deviation $\sigma_n$,

$$p(x\mid y)\propto\exp\!\left[-\frac{\lVert y-x\rVert^2}{2\sigma_n^2}\right]p(x).$$

The likelihood favors agreement with the captured image, while the image prior assigns probability to possible clean structures. A conditional generative model can be trained using paired data. Alternatively, a pretrained diffusion prior can be combined with a measurement model during sampling. DDRM treats linear inverse problems (Kawar et al. 2022 (DDRM)); diffusion posterior sampling uses approximations to handle more general noisy inverse problems (Chung et al. 2023 (DPS)). These practical algorithms should not be assumed to draw exact samples from the true camera posterior.

Repeated runs with different random seeds reveal variation allowed by the model and sampler. Their empirical mean approaches the model's conditional mean if the samples correctly represent that distribution. A finite average remains noisy, and biased or poorly mixing samplers can miss alternatives. Record the seed, model weights, sampler and precision for reproducibility; a seed alone does not fix nondeterministic kernels or a changed implementation.

Sampling usually requires repeated network evaluations. Report their number and the actual latency along with quality. A comparison that selects the best of many samples using the clean reference has given the method information unavailable at deployment.

13.9.3 The reverse process is a sequence of denoisings⧉

A common diffusion training construction corrupts a clean image $x_0$ at a chosen level $t$:

$$z_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon, \qquad \varepsilon\sim\mathcal N(0,I).$$

The schedule coefficient $\bar\alpha_t$ specifies the retained signal power. A network predicts $\varepsilon$ from $z_t$ and $t$ using squared error. Its prediction gives a clean-image estimate

$$\hat x_0=\frac{z_t-\sqrt{1-\bar\alpha_t}\,\varepsilon_\theta(z_t,t)}{\sqrt{\bar\alpha_t}}.$$

The reverse update uses this prediction and the schedule to construct the next state (Ho et al. 2020). It is not simply repeated replacement of an image by a conventional denoiser output. Stochastic samplers add prescribed noise; deterministic formulations use a different update. Measurement-conditioned sampling must incorporate $y$ consistently with the chosen formulation. Chapter 7.4, Generative AI and diffusion develops those algorithms.

For $\bar\alpha_t>0$, rescaling $z_t$ by $1/\sqrt{\bar\alpha_t}$ yields an additive-noise problem with effective standard deviation $\sqrt{(1-\bar\alpha_t)/\bar\alpha_t}$. This relates diffusion time to a noise-level input; the two are connected by a parameterization rather than numerically identical. Chapter 13.11, Denoising as a universal prior derives the exact Gaussian posterior-mean/score identity and describes a simple score-based sampling iteration. Denoiser-based generation also predates some diffusion restoration methods (Kadkhodaie & Simoncelli 2021).

13.9.4 What a hallucinated detail costs⧉

Plausible texture can be useful in an image intended for visual presentation. It is less useful when a user needs to know whether a small feature was actually present. The same issue applies to ordinary denoisers that remove unusual details; generative reconstruction can additionally introduce sharply resolved alternatives.

Sample disagreement can show where a model offers different reconstructions. It is not automatically a calibrated uncertainty map. A model may assign negligible probability to the truth, or a sampler may repeatedly visit only one plausible explanation. Pixelwise variance also fails to summarize all structural alternatives.

Forward-model residuals provide a complementary check. For a reconstruction $\hat x$, inspect $y-A\hat x$ in the domain and units of the measurement noise. A noisy observation generally should not be fitted exactly. Residual magnitudes and structure can expose a mismatch, but a plausible reconstruction can remain consistent with the measurement and still be wrong. Part 41, Image Forensics and Authentication discusses the consequences when reconstructed details are treated as evidence.