13.13 The limits of denoising⧉
Denoising has limits set by the measurements and the assumed image distribution. Levin and Nadler Levin & Nadler 2011 and Levin et al. Levin, Nadler, Durand & Freeman 2012 study bounds and estimates under specified statistical and patch assumptions. They do not provide a universal numerical floor independent of those assumptions. More photons or additional independent measurements can reduce ambiguity; a stronger prior can resolve it differently, sometimes by imposing incorrect detail.
One caveat about these bounds: they are stated in terms of mean squared error (an $L_2$ distance to the clean image). That makes the result rigorous, but it also limits what it tells us: an $L_2$-optimal denoiser is not the same as a perceptually optimal one. A result the bound calls "as good as possible" by MSE may still look improvable to the eye, or to a learned perceptual metric (LPIPS and friends, from Chapter 9.3, Image metrics); and a learned or generative denoiser that hallucinates plausible texture can score worse on $L_2$ while looking far better. So read the Levin–Nadler floor as a sharp statement about information and squared error, not as the last word on perceived quality.
Under squared-error loss the optimal estimate is the posterior mean, $\hat{\mathbf x}=E[\mathbf x\mid\mathbf y]$. If an edge could lie at several nearby positions, averaging those possibilities spreads it out. A sharper posterior sample can look more plausible while disagreeing more with the recorded scene. Report reconstruction error and appearance separately.
Squared-error minimization returns the posterior mean. When plausible answers disagree, their average can blur edges or desaturate uncertain colors. A perceptual or generative objective can produce a sharper alternative, but sharpness does not establish fidelity. Evaluate both properties.
Push denoising far enough, especially on skin, and it stops removing noise and starts erasing texture: pores, fine lines, the micro-detail that makes skin read as skin. The result is the familiar plastic or waxy look, and at that point a noise filter has become a beauty filter. The boundary is blurry because skin smoothing, blemish removal and beautification are one operation, suppressing high-frequency variation in regions a face or skin detector has flagged, set to different strengths. Some phone camera applications have enabled skin smoothing by default, often on the front camera. That default applies one standard of flawless skin without the subject choosing it, and the photograph becomes less faithful to the person in front of the lens. Keep the original, make the strength a deliberate and visible choice, and decide where your pipeline draws the line between cleaning an image and editing a face; Chapter 40.2, Ethics of computational photography discusses the broader implications.
There are two separate reasons for that mismatch. First, plausible clean images do not form a convex set: averaging two plausible alternatives need not give an equally plausible photograph. Blend a sharp edge at one position with the same edge one pixel away and the result can have a softened or doubled transition. The posterior mean averages over such alternatives because that minimizes expected squared error, even when no one alternative contains the resulting blur.
Second, pixelwise squared error is not a perceptual distance. A small displacement of a detailed image can incur a large error even when the scene remains recognizable. Different arrangements of errors with the same MSE can have different visibility. The geometry of the plausible alternatives and the choice of loss therefore pose distinct problems. A perceptually convincing output may still choose the wrong edge position or invent the wrong texture.
Recap: big lessons of this chapter
Squared-error minimization returns the posterior mean. When plausible answers disagree, their average can blur edges or desaturate uncertain colors. A perceptual or generative objective can produce a sharper alternative, but sharpness does not establish fidelity. Evaluate both properties.