BookNotesSlidesP-sets Syllabusdraft · v0.2.49.1
◀▲⌂▶
💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.

13.7 Learned denoisers⧉

Wavelet shrinkage assumes that useful image structure has a sparse representation. BM3D Dabov et al. 2007 searches for repetition within the observed photograph. A learned denoiser fits a mapping from noisy images to clean targets, using regularities across a training collection as well as spatial context within each input. Its behavior depends on the training images, corruption model, architecture, and loss.

For paired training with squared error, the ideal population solution is the posterior mean:

$$\arg\min_f\mathbb E\lVert f(y)-x\rVert_2^2=\mathbb E[x\mid y].$$

This characterizes an unrestricted optimum. Finite networks, finite data, and imperfect optimization only approximate it. A network can also be trained with another loss, in which case the optimal statistic changes. These distinctions matter when a trained denoiser is later interpreted as a score estimator in Chapter 13.11, Denoising as a universal prior.

13.7.1 DnCNN: learn the noise, not the image⧉

DnCNN (Zhang et al. 2017 (DnCNN)) predicts a residual that is subtracted from the input:

$$\hat x=y-\mathcal N_\theta(y),\qquad \mathcal L(\theta)=\mathbb E\big\lVert\mathcal N_\theta(y)-(y-x)\big\rVert_2^2.$$

The residual target is the noise added to each training example. Under a white Gaussian model it has flat expected spectral power, including at low frequencies. Residual learning does not work by making those frequencies disappear. It provides an identity path from input to output, so the network parameterizes a correction instead of having to reproduce the whole image. Substituting $\hat x=y-\mathcal N_\theta(y)$ shows that this loss is algebraically the same squared reconstruction error.

At the population optimum the network predicts $\mathbb E[n\mid y]$, not the particular unknown noise realization with certainty. The removed residual may therefore contain real structure, and the output may retain noise even when training and test distributions match. Inspecting the residual is useful for detecting texture loss, but a structureless residual is not a certificate of correct denoising.

The original architecture uses convolutional layers, rectified activations, and batch normalization without spatial downsampling. Fixed-level and blind Gaussian versions illustrate two choices: supply a model trained for a specified noise level, or train across levels and infer the necessary correction from the image. Generalization beyond the tested noise range and image domain must be measured.

fig-residual-denoise
Figure 13.7.1. Interactive. The residual shown as an image, beside the input and the result, for a denoiser tuned to one noise level while the test noise is swept across it. The denoiser is wavelet shrinkage with its threshold fixed at the training level, a stand-in for a trained network, which the book cannot ship; the failure it shows has the same cause, an assumption about the noise level that no longer holds. At the matched level the residual is close to pure noise, which is what a well-matched denoiser leaves. The two mismatches fail in opposite directions, and the residual names which one you are in. Below the training level the threshold sits far above the noise and cuts into the signal, so edges and texture appear in the residual. Above it the threshold sits far below the noise, so the filter under-filters, the residual stays noise, and the surviving noise is in the output instead; the panel reports how strongly the residual correlates with the scene in each case. Structure in the residual means too much was removed; noise in the output means too little was. Interactive: set the test noise level and watch it cross the training level, and toggle between the result, the residual and a difference against the clean reference.

13.7.2 FFDNet: hand the network the noise level⧉

FFDNet (Zhang, Zuo & Zhang 2018 (FFDNet)) accepts a noise-level map along with the image. It first rearranges neighboring pixels into channels at a lower spatial resolution, performs convolutional processing with the map, and rearranges the output back. This reversible rearrangement expands the effective receptive field without discarding samples.

Write a conditioned denoiser as $\hat x=D_\theta(y,\sigma(\cdot))$. A constant map requests one noise level throughout; a varying map describes spatial variation. For the calibrated approximation $\operatorname{Var}(n_p\mid x_p)=ax_p+b$ from Chapter 13.1, Noise models, transforms, and benchmarks, one can estimate a map using a pilot clean estimate. Supplying the correct variances does not by itself make an independently Gaussian-trained network correct for correlated, clipped, or non-Gaussian noise.

Check the model interface before constructing this map: a standard deviation map and a variance map differ by a square, and a level in 8-bit units differs by 255 from a unit-range level. The map also exposes a useful control: asking for a larger level generally increases smoothing. Its usable range remains a property of the trained model. Camera calibration helps determine the input level, but transformations, color gains, and the unknown underlying signal still have to be accounted for. An arbitrary processed image requires separate noise estimation or a suitable blind model.

13.7.3 DRUNet: a denoiser built to be reused⧉

DRUNet combines a U-Net's multiscale processing and skip connections with residual blocks and a noise-level input (Zhang et al. 2021 (DRUNet, deep plug-and-play)). The encoder reduces spatial resolution to gather broader context; the decoder combines that context with features retained at finer scales. The model was developed as a denoising component for plug-and-play image restoration Venkatakrishnan et al. 2013.

In such a solver the supplied level controls a prior step on an intermediate image. It need not equal the sensor noise in the original measurement. A level-conditioned network allows this strength to change across iterations without loading a separate network for every setting. Smooth-looking changes in its output do not, however, prove convergence of the surrounding solver.

The published DRUNet training uses an $L_1$ reconstruction loss. Its ideal scalar predictions are conditional medians, whereas the squared-error result above gives conditional means. DRUNet can work well as a practical restoration prior without satisfying the exact posterior-mean assumption used in Tweedie's formula Efron 2011 (Tweedie’s formula).

13.7.4 Unrolling: from learning a function to learning an algorithm⧉

An iterative solver can also supply the structure of a network. Consider sparse coding with dictionary $B$, coefficient vector $z$, and objective

$$\min_z\frac12\lVert Bz-y\rVert_2^2+\lambda\lVert z\rVert_1.$$

ISTA Daubechies, Defrise & De Mol 2004 (ISTA) alternates a data-gradient step with elementwise soft thresholding:

$$z^{k+1}=\operatorname{soft}_{\alpha\lambda} \left[z^k-\alpha B^{\mathsf T}(Bz^k-y)\right].$$

Here $B$ maps coefficients to image samples, even for a denoising problem whose image-domain forward operator is the identity. The step $\alpha$ scales both the gradient and threshold. For example, $0<\alpha\le1/\lVert B\rVert_2^2$ is a standard choice for convergence of this convex problem.

LISTA (Gregor & LeCun 2010 (LISTA)) replaces a chosen number of iterations by learned layers of the form

$$z^{k+1}=\operatorname{soft}_{t_k}(W_y^{k}y+W_z^{k}z^k).$$

ISTA corresponds to $W_y^{k}=\alpha B^{\mathsf T}$, $W_z^{k}=I-\alpha B^{\mathsf T}B$, and $t_k=\alpha\lambda$. Learning can untie these quantities, and an implementation can share some weights across layers. The output $Bz^T$ is trained or evaluated against the chosen reconstruction target.

Algorithm 13.7.1
z = zeros(number_of_coefficients)
for k in 0..T-1:
    z = soft_threshold(W_y[k] @ y + W_z[k] @ z, threshold[k])
xhat = B @ z

A finite learned network can reach a useful approximation in fewer steps than its source algorithm on the training distribution. It need not minimize the original objective, and repeating learned layers beyond their trained depth need not converge. Parameter constraints can preserve some solver properties, at a cost in flexibility. ISTA-Net learns transform operations for compressive reconstruction (Zhang & Ghanem 2018 (ISTA-Net)); unrolling more generally includes learned variants of splitting and other iterative methods (Monga, Li & Eldar 2021 (algorithm unrolling)).

fig-unrolling-diagram
Figure 13.7.2. Interactive. An iterative algorithm and a network drawn as the same object. The left panel runs ISTA on a small problem with a slider for the iteration count, showing the estimate at each step; the right panel is the same steps drawn as layers, with the parameters that unrolling makes learnable highlighted. Switch the learned parameters on and the layer count needed for the same accuracy drops, which is the entire claim of the method. Here the two learnable parameters, the step size and the threshold, are fitted by a small search on the same problem the error is measured on, so the drop shows what fitting can buy, not that the result generalizes to new problems. Interactive: set the iteration or layer count, toggle learned against fixed parameters, and read the reconstruction error against step count for both.

Unrolling starts with the steps of an iterative method rather than an arbitrary layer stack. Write a fixed number of iterations as a computation graph, then allow selected matrices, transforms, step sizes, or thresholds to be fitted from examples. Different iterations can learn different parameters, spending a limited depth differently from the original convergent solver. LISTA is an early example Gregor & LeCun 2010 (LISTA). Its attraction is the connection between a useful algorithmic structure and a learned finite computation. Its output need not be the exact minimizer of the objective that motivated the original iteration.