BookNotesSlidesP-sets Syllabusdraft · v0.2.49.1
◀▲⌂▶
💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.

13.0 DENOISING⧉

A short exposure in a dim room preserves motion but leaves visible noise. We can reduce that noise by combining several frames, averaging similar regions within one photograph, or using a model learned from other photographs. Each approach needs a reason to treat some variation as noise and preserve the rest as signal.

Chapter 2.8, Noise, signal-to-noise ratio and dynamic range established the sensor model, the distinction between noise and bias, and the improvement from collecting more light. Here we use that model to reconstruct an image. For denoising, the observation is $y=x+n$: the desired image $x$ is measured directly, with an error $n$. Here $x$ is the clean image at the captured resolution, including the optical blur already present. Denoising does not undo that blur; a single noisy value still does not determine its clean value.

An explicit prior expresses which candidate images we prefer. Under independent Gaussian noise of constant standard deviation $\sigma$, one common estimate is

$$\hat x=\arg\min_x\frac{1}{2\sigma^2}\lVert x-y\rVert_2^2+\lambda\Phi(x).$$

The first term penalizes disagreement with the measurement, measured relative to its noise level. The second penalizes images disfavored by the prior $\Phi$; $\lambda$ controls its influence. For example, a smoothness penalty discourages rapid changes between neighboring pixels. It removes noise in a wall but can also remove the weave of a fabric. A signal-dependent or correlated noise model requires a corresponding likelihood, rather than this constant-variance squared error.

A prior supplies assumptions, not additional observations of the scene. Repeated texture in the same image can be useful evidence, but copying it to another location is still an inference. A learned model may produce plausible detail that was never resolved by the camera. We will distinguish agreement with the measurements, accuracy against a reference, and visual plausibility when judging the result.

This part also opens a larger family of reconstruction problems. Denoising deals with uncertain measured values; super-resolution and demosaicking deal with spatial or color samples that were not directly recorded; deblurring deals with information attenuated, and sometimes erased, by a blur. Inpainting supplies missing regions. Matting and intrinsic-image methods separate contributions mixed into one observation. These tasks differ in their forward models and ambiguities, but all require us to distinguish what the data constrains from what an assumption supplies.

This organization is useful because an apparently unrelated method can reuse the same inference machinery. A new forward operator changes the data term. A different image model changes the prior. Neither change removes the need to state what was measured. Copying a repeated patch within an image, for example, uses real samples but still assumes that the copied structure belongs at its new location.

13.0.1 The second route: more than one frame⧉

If the scene can be photographed repeatedly, each frame supplies another measurement. For aligned captures with unchanged settings and independent, zero-mean noise, averaging $N_f$ frames reduces noise standard deviation by $\sqrt{N_f}$. It needs no learned image prior, but it does require that corresponding pixels measure the same scene value. Motion, occlusion, clipping, and calibration errors can violate that condition.

Chapter 13.2, Denoising by averaging develops the estimator and its practical limits. Related capture methods vary the measurement: exposure brackets record different brightness ranges, focal stacks record different focus distances, and shifted frames can provide extra spatial samples. Their reconstruction must account for those differences before combining values.

Multiple captures also defer a decision. A bracket records alternatives in exposure, a focal stack records alternatives in focus, and a burst records repeated samples that can be selected or combined later. The photographer still makes a capture choice, but need not settle every tradeoff in one exposure. The computation can be inspected and rerun after the moment has passed.

Two operations recur across those applications: alignment establishes which samples can be combined, and robust combination limits the influence of samples that do not fit. A moving subject in an HDR bracket and an outlier in an averaging stack are different causes of the same practical danger, combining values that do not represent the intended scene quantity. The appropriate rejection rule depends on the noise and motion model.

Local smoothness, sparse transform coefficients, and repeated patches supply different priors for single-image denoising. Learning fits image regularities from examples; self-supervision changes the required training targets. Each choice constrains what can be recovered from a given noisy input.

Chapter 13.11, Denoising as a universal prior explains how a denoiser can be reused inside other reconstruction algorithms. This reuse has conditions: an arbitrary denoiser need not define an explicit penalty or produce a convergent iteration. The connection between a Gaussian denoiser and the score of a noisy-image distribution also explains its role in generative modeling.

Part 14, Super-resolution and upscaling introduces blur and subsampling into the observation model. Part 15, Demosaicking and joint methods treats the missing color samples of a sensor mosaic and the interaction between reconstruction stages. Joint estimation can avoid propagating errors from one stage into the next, although its accuracy still depends on the model and training data.

For a linear forward operator $A$, the corresponding Gaussian data term is $\lVert Ax-y\rVert_2^2/(2\sigma^2)$. Some operators attenuate information; others erase it. A reconstruction method must respect that distinction when deciding how strongly to rely on the prior.

Contents of this part

▸ full collapsible outline of this part