7.2 Machine learning🔗⧉

To train a denoiser, collect noisy measurements and the clean images it should predict. Fit a function to those pairs and evaluate it on new scenes. The model’s architecture matters, but so do the cameras, degradation process, target definition, and loss represented by the data. Refreshers#Machine learning and deep learning covers the training machinery; here we examine those imaging choices.

The idea sits above any particular model. You could learn an operator with a linear model, a decision tree or a large network, and imaging mostly uses networks. Chapter 7.3, Deep learning covers the concrete network operators. This chapter covers what it means to learn an operator instead of designing it, and why the data (how it is generated, how faithfully it models the camera's noise, which dataset is used) is as much part of the method as the model.

7.2.1 The framing: learned operators replace hand-designed ones🔗⧉

Every recovery task in this part so far arrived with a hand-designed operator. Deblurring used an inverse filter; denoising leaned on a smoothness prior; dehazing reached for a dark-channel heuristic; demosaicking interpolated the missing colors with a carefully tuned rule. In each case a person looked at the problem, decided what made a good answer, and wrote that decision down as a formula. This chapter is about an alternative to those hand-designed maps: instead of writing the operator, fit it to data. Collect a pile of example pairs, with a degraded input next to the desired answer, and let a function with millions of free parameters adjust itself until it reproduces those answers (Figure 7.2.1⛶ fullscreen). The function is typically a neural network; we write it $f_\theta$, where $\theta$ is the bag of parameters that training sets. The recovered image is then just $\hat I = f_\theta(\text{measurement})$, and training is the optimization

over a dataset of input–target pairs $(x_i, y_i)$: search over parameters $\theta$ for the ones that make the operator's output $f_\theta(x_i)$ land closest, under a loss $\ell$, to the desired target $y_i$ on every example. That is the whole framing. The main choices are what network to use (deferred to Chapter 7.3, Deep learning and the refresher) and where the pairs $(x_i, y_i)$ come from (the second half of this chapter).

Learning can replace a penalty inside an explicit reconstruction method, or train the whole map from measurement to estimate. These are different designs. A learned penalty can retain an explicit data term; a direct predictor $f_\theta$ need not minimize any stated energy or enforce measurement consistency. Its behavior combines the training distribution, loss, architecture, and optimization. Test $Af_\theta(y)$ against the observation when the forward model is known.

Why did this take over, and why now? This change depended on large datasets, fast graphics processing units (GPUs), and reusable building blocks. With those in hand, fitting a map that once required extensive hand-tuning became much faster. There is a one-line argument behind the trend, Sutton's Bitter Lesson: general methods that scale with data and compute tend, eventually, to beat methods that bake in human cleverness. We will see that play out task by task in the next chapter, always with the costs in view: data hunger, compute, and hallucinated detail.

Split data by scene or capture sequence before extracting patches. Otherwise neighboring crops or burst frames can leak into both training and evaluation. Use validation data to choose losses, hyperparameters, and stopping; reserve a separate test set for the final comparison. Include cameras, exposure levels, and content absent from training when testing transfer.

split reference scenes into train, validation, test
for each training step:
    target = sample_reference_patch(train)
    settings = sample_degradation_parameters()
    measurement = simulate(target, settings)
    prediction = model(measurement, known_settings(settings))
    update parameters using loss(prediction, target)
select checkpoint using validation data
freeze model; evaluate on held-out synthetic and real captures

Supply only settings available at deployment. Normalize loss and image ranges consistently, and compare with a simple baseline under the same preprocessing.

7.2.2 The data story: synthetic data, noise models, datasets🔗⧉

Beyond L7.6show, one point matters most: the data is the method. A learned operator is exactly as good as the pairs it trained on, and good pairs are usually not available. They have to be generated. The architecture gets the headlines, but two networks of the same shape, one trained on careless data and one on faithful data, are not the same tool at all. So before we ever pick a model, we have to ask where the training pairs come from.

Generating synthetic data. The trick is to start from clean targets and simulate the degradation to produce (input, target) pairs at scale (Figure 7.2.3⛶ fullscreen). This is often the only way to get supervised data at all: accurately aligned clean and noisy captures can be expensive or impossible for moving scenes, so you take a clean image and add the corruption yourself: you know the answer because you started from it. And here computational photography is unusually lucky: because we know the forward model, the physics of how a scene becomes pixels, many degradations can be approximated from reference images. Depth-dependent blur, occlusion, and newly visible surfaces require additional scene information. The menu is essentially the whole image-formation pipeline run forward:

• noise: add a realistic signal-dependent model (Poisson shot noise plus read noise; variance is approximately affine in linear signal), using a constant Gaussian approximation only where appropriate (the crux, below);
• clipping: saturate the highlights (and crush the blacks) the way a real sensor does, so the network learns to cope with lost information rather than assuming a clean linear range;
• camera shake: not one fixed kernel but an integral over a family of homographies (known camera rotation during the exposure; translation also depends on depth), so the synthetic blur is spatially varying under that motion model, as the deblurring chapter insists;
• mosaic: sample through a Bayer (or X-Trans) color-filter array, optionally modeling pixel cross-talk between neighbors, to make demosaicking pairs;
• defocus / lens blur: convolve with a point-spread function, ideally one from an actual optical simulation (ray-trace the lens) rather than a tidy disk, so aberrations and field dependence are present;
• downsampling: shrink a sharp image (through a proper prefilter) to mint super-resolution pairs, where the high-frequency target is, by construction, known.

The reference is exact ground truth for the simulation. It may already contain sensor noise, sharpening, compression, or demosaicking artifacts, which the network can learn to reproduce. A differentiable simulator is useful when optimizing camera or degradation parameters through it; generating supervised pairs does not itself require differentiability.

Levels of supervision. Not all learning needs perfectly paired data, and the methods sort by how much labeling they demand. Fully supervised learning wants exact (input, target) pairs, exactly what the synthetic pipeline above manufactures. Weakly or loosely supervised learning makes do with cheaper signals, such as an image-level tag instead of a per-pixel mask or an unpaired set of "good" images rather than matched pairs (the CycleGAN regime), or a physical constraint the answer must satisfy. Self-supervised learning invents its labels from the data itself. For example, Noise2Noise denoises from noisy images alone, masked-prediction pretraining hides part of an image and predicts it, sidestepping clean targets entirely. And generative training learns the distribution of natural images rather than a single right answer, so it can sample a plausible output where a supervised model would only average (the engine behind diffusion, and the cure for the dull-mean problem of Part 13, Denoising]). Most real systems mix these: a physics-simulated supervised core, a self-supervised backbone, and a generative prior for the parts the data cannot pin down.

A model trained on constant-variance Gaussian noise can perform poorly when deployed on signal-dependent raw noise or correlated processed noise. Shot-noise variance follows the photon count; read noise can dominate at low signal. Demosaicking, resampling, and compression change the distribution and correlations (Chapter 2.8, Noise, signal-to-noise ratio and dynamic range). Match the intended raw or rendered domain and camera settings. Hard-case mining in Gharbi et al. 2016 and the degradation model in Wang et al. 2021 address different aspects of training-data coverage.

An ISP simulator should respect operation order: optical integration and photon detection precede read noise, clipping, demosaicking, tone mapping, and compression. Approximate “unprocessing” of a rendered image can produce useful synthetic raw data, but cannot invert lost highlights, quantization, or unknown local edits. FlexISP Heide et al. 2014 instead demonstrates joint reconstruction with a modular image-formation model; it is not an exact inverse for arbitrary camera JPEGs.

Standard datasets and benchmarks. Know the workhorses and what each measures, because the benchmark quietly defines the task: ImageNet (pretraining and features) Deng et al. 2009 (ImageNet), DIV2K (super-resolution), SIDD and DND (real-world denoising), MIT-Adobe FiveK (retouching), KITTI and NYU-Depth (depth), COCO (detection and segmentation). The MIT-Adobe FiveK set is a clean illustration of the point. It contains the same scenes retouched by five expert editors, which is exactly what made a photographer's taste learnable in the first place. And a benchmark's biases become the model's biases: a depth dataset that is all indoor rooms will leave a model confused outdoors, so a dataset is never neutral. The choice of benchmark is a design decision as consequential as the choice of architecture.

Noise2Noise Lehtinen et al. 2018 uses aligned noisy pairs with the same underlying signal. Under squared loss, the target corruption must have zero conditional mean given the signal and input. Noise2Void Krull et al. 2019 hides the target pixel and predicts it from context, relying on noise independence across the hidden pixel and its context. Correlated noise, clipping, or alignment errors can break these conditions. Training without clean targets removes one data requirement, not the need for assumptions; the denoising chapters give the details.

Generating the input from a clean target makes the pairing easy to control. Blur the target with a known kernel, select a known sampling grid, or draw noise from a specified model, and the clean array remains available as the training answer. The hard part is whether that operation resembles the camera. A denoiser trained on equal-variance noise can learn the wrong strength of smoothing when sensor variance rises with signal. A model trained on an ideal resize can learn edges that fail after a different resampling or compression pipeline. Synthetic pairs remove a labeling problem while creating a modeling obligation.