13.8 Denoising without clean data⧉
A clean reference for a real noisy photograph is difficult to capture. Averaging reduces reference noise but requires alignment and scene stability. Changing exposure or gain changes the noise statistics and may introduce motion or clipping. Such pairs can be collected, as SIDD Abdelhamed, Lin & Brown 2018 (SIDD) and DND Plötz & Roth 2017 (DND) demonstrate, but a large, diverse training set requires considerable care.
Self-supervised denoising replaces clean targets with noisy measurements and assumptions about their relationship. The central requirement is that the target noise cannot be predicted from the information given to the estimator. Signal-dependent shot noise can satisfy this requirement in an appropriate raw model. Shared pattern noise or processing-induced correlations can violate it. Independence and signal dependence are different properties, as Chapter 2.8, Noise, signal-to-noise ratio and dynamic range explains.
13.8.1 CBDNet: estimate the noise, keep the pairs⧉
CBDNet (Guo et al. 2019 (CBDNet)) is a supervised response to the mismatch between synthetic white noise and real photographs. It combines a noise-estimation subnetwork with a conditioned denoiser, models signal-dependent noise and camera processing in synthetic training pairs, and also uses real pairs.
An asymmetric loss penalizes underestimating the noise level more strongly than overestimating it. This encodes a preference about residual noise versus detail loss; it is not a consequence of the physical noise model. CBDNet is a useful historical example of improving the training distribution while retaining paired supervision. The quality of its output still depends on how well that distribution describes the test camera and processing pipeline.
13.8.2 Noise2Noise: two noisy observations⧉
Noise2Noise (Lehtinen et al. 2018) replaces each clean target with another noisy observation of the same underlying signal. Let $y_1=x+n_1$ and $y_2=x+n_2$. A sufficient condition for squared-error training is
Conditionally independent captures with conditionally zero-mean target noise satisfy it. Expanding the loss gives
because the cross term has zero expectation. The final term does not depend on $f$, so the population minimizer is unchanged. Equivalently,
Noisy targets increase finite-sample uncertainty; the equality does not promise identical training results from the same number of pairs. Nor does it say that the network recovers each clean image exactly. It learns the conditional mean supported by its input and training distribution.
Pairs of this kind are easier to find than clean images. A microscope can image the same fixed specimen twice; a telescope records many exposures of the same field; a camera on a tripod can shoot a static scene repeatedly; consecutive frames of a video of a static scene serve too. In each case the second capture is a candidate target, not an automatically valid one. Two frames from the same sensor can share fixed-pattern errors. Clipping can make noise nonzero-mean. Motion, illumination changes, or inaccurate alignment can make the target describe a different signal. Even registration estimated from noisy frames may couple input and target errors. These effects must be checked rather than treating a burst as automatically valid Noise2Noise data. When the assumptions hold, signal-dependent target variance is allowed.
13.8.3 Noise2Void and Noise2Self: one noisy image⧉
If only unpaired noisy images are available, neighboring pixels can predict the signal at a withheld pixel. A blind-spot predictor $f_i(y_{-i})$ does not observe $y_i$. Under the condition $\mathbb E[n_i\mid x,y_{-i}]=0$, its noisy-target squared loss differs from its clean-target loss by a constant, just as above.
The ideal predictor is $\mathbb E[x_i\mid y_{-i}]$. It uses less information than $\mathbb E[x_i\mid y]$: an isolated feature visible only at the center may be impossible to infer from its neighbors. Spatial predictability of the scene determines how much this exclusion costs.
Noise2Void uses masking to train such predictions (Krull et al. 2019). Noise2Self formalizes predictors that exclude a subset $J$ when estimating that subset, called $\mathcal J$-invariance (Batson & Royer 2019 (Noise2Self)). A compatible independence assumption must hold between each withheld subset's noise and the information used to predict it. This also permits selecting denoising parameters by noisy-data loss within the admissible predictor class.
patch = sample_patch(noisy_training_images) J = choose_masked_positions(patch) inp = patch.copy() for i in J: inp[i] = replacement_from_unmasked_neighbors(patch, i, J) pred = net(inp) loss = mean((pred[J] - patch[J])**2) update_network(loss)
Compute loss only at withheld positions. Replacements must exclude the target itself and must not copy another withheld target into an input location that reveals it. At an image boundary, test exclusion using the actual sampled index: a clamped neighbor can be the target even when its nominal offset is nonzero. A structural blind-spot network can enforce exclusion directly. Merely training a normal network to reproduce its unmasked input permits the identity solution. Training can use many unpaired noisy images; learning from patches of a single photograph is possible but supplies less diversity and no universal quality guarantee.
13.8.4 Blind2Unblind: recovering information hidden by the blind spot⧉
Blind2Unblind addresses the information discarded by blind-spot training (Wang et al. 2022 (Blind2Unblind)). Its global-aware mask mapper collects predictions at masked positions, and its re-visible loss relates masked predictions to an unmasked network pass while controlling the identity-mapping problem. It is not simply a larger blind spot that makes arbitrary correlated noise independent.
Correlation remains a separate concern. Demosaicking, resampling, and sharpening mix neighboring measurements; compression can couple errors over blocks. A withheld pixel's noise may then be predictable from visible neighbors. Pointwise lens-shading multiplication changes variance but, by itself, does not create spatial correlation from independent inputs.
A larger exclusion region can reduce leakage when correlations are sufficiently local, at the cost of losing more scene information. Shared row offsets and other long-range components may remain. A real-noise benchmark result supports performance on that benchmark; it does not establish the noisy-target identity for every processed image.
There is a general lesson here, and it applies well beyond denoising. A self-supervised scheme replaces some missing supervision with assumptions about the data. Violating those assumptions can defeat an otherwise suitable architecture. When such a method underperforms on real data, check the assumptions alongside optimization and training data.
13.8.5 Choosing supervision for a real camera⧉
Choose supervision by examining both the available captures and their noise statistics:
| available data | possible approach | condition to check |
|---|---|---|
| aligned noisy/clean pairs | supervised denoising | reference noise, registration, clipping, and domain coverage |
| two noisy observations per scene | Noise2Noise | same underlying signal and target noise with zero conditional mean |
| unpaired noisy images | blind-spot self-supervision | target noise unpredictable from visible pixels; enough spatial signal context |
| calibrated raw model and representative clean images | synthetic supervised training | simulator matches the sensor and processing used at test time |
These are alternatives, not a ranking by data quantity. Raw data often makes the model easier to characterize, but can still contain correlated read noise, defects, and clipping. For an unknown processed photograph, a pretrained real-noise model may be useful, while its applicability remains an empirical question. Compare methods on held-out camera data when possible; neither self-supervision nor calibration alone removes a distribution mismatch.