14.6 The perception-distortion tradeoff and perceptual super-resolution🔗⧉

Two questions get asked of a restored image and they are not the same question.

The first is how close it is to the truth. Take the output, take the original, and measure a distance: squared error, PSNR, SSIM, a learned feature distance (Zhang et al. 2018). This is distortion, it is defined per image, and it needs the ground truth.

The second asks whether outputs have the statistical characteristics of real photographs. The formal perception criterion below compares distributions. It differs from a viewer judging one crop or a full-reference feature metric comparing two images.

For ambiguous reconstruction problems, pushing expected distortion toward its minimum can worsen distributional realism. A method can still improve both measures if the previous method was suboptimal.

14.6.1 The tradeoff, stated🔗⧉

Let $p_X$ be the distribution of natural high-resolution images and $p_{\hat X}$ the distribution of a method's outputs. Define perceptual quality as a divergence $d(p_X, p_{\hat X})$, small when the outputs are distributed like real photographs, and distortion as an expected per-image distance $\mathbb{E}[\Delta(X, \hat X)]$. Blau and Michaeli's result (Blau & Michaeli 2018) is that the achievable region of the plane is bounded by a curve

The optimum $P(D)$ is non-increasing because a larger distortion budget permits every estimator available under a smaller one. Convexity additionally requires a divergence convex in the output distribution and an admissible class that permits randomized estimators. The curve can have flat regions. These statements do not require every pair of practical methods to trade one metric against another.

Under squared error, the conditional mean $\mathbb E[X\mid Y]$ minimizes expected distortion. If uncertainty remains, averaging alternative textures or edge positions can change the output distribution. It need not make every output visibly blurred; when $Y$ determines $X$, the conditional mean recovers it exactly and no such conflict remains. Minimum expected MSE also differs from maximum expected PSNR because PSNR contains a logarithm.

For practical comparison, distinguish the chosen operating point from the quality of the method.

Methods at different operating points can be compared, but a single winner depends on the application. A useful comparison matches one measured axis and examines the other, or presents both over a parameter sweep. Measured points need not lie on the theoretical frontier.

A scalar utility can rank methods for a specified use. Reporting its component measures makes that preference visible and permits other applications to choose differently.

14.6.2 Buying perceptual quality: perceptual losses and adversaries🔗⧉

Perceptual training changes the objective. SRGAN (Ledig et al. 2017 (SRGAN)) used feature content and adversarial losses. The three-term expression below is a common generic combination, not a claim that every cited method used these exact terms and weights.

A content loss compares features of the output and reference (Johnson et al. 2016). Its tolerance to shifts or texture changes depends on the feature extractor and layer; it does not simply stop penalizing one-pixel misalignment.

An adversarial loss trains the generator against a discriminator. It encourages output statistics resembling training photographs, but a finite discriminator need not measure the true distribution divergence or establish scene fidelity.

A pixel reconstruction loss compares with the high-resolution training target. It is different from a measurement-consistency term comparing the degraded output with the observed low-resolution input.

# perceptual super-resolution training step, the shape of it
xhat = G(y)
loss_content = mse(vgg_features(xhat), vgg_features(x))
loss_adv     = bce(D(xhat), REAL)                  # generator wants D fooled
loss_pix     = mse(xhat, x)
step(G, loss_content + lam_adv*loss_adv + lam_pix*loss_pix)
step(D, bce(D(x), REAL) + bce(D(xhat.detach()), FAKE))
# loss weights are tuned on validation data for the intended fidelity/appearance tradeoff

Tune $\lambda_{\text{adv}}$ and the other weights on validation data. They affect optimization and output quality; changing them does not guarantee movement along the optimal frontier.

ESRGAN (Wang et al. 2018 (ESRGAN)) refined the architecture and used a relativistic discriminator, which judges whether an image is more realistic than a real one rather than realistic in absolute terms, and it removed batch normalization for the artifacts it caused. Real-ESRGAN (Wang et al. 2021) kept the losses and changed the training data, which is Chapter 14.9, Blind and real-world super-resolution's subject and the change that made these methods usable on real photographs.

14.6.3 Measuring on both axes🔗⧉

Report reference error (such as PSNR), a suitable full-reference similarity measure such as LPIPS or DISTS, and representative images under a common display transform (Zhang et al. 2018; Ding et al. 2020 (DISTS)). These metrics assess different properties and do not replace the distributional divergence in the theorem.

Without a reference, use blinded visual comparison, consistency with the assumed measurement model, and clearly identified no-reference metrics such as NIQE (Mittal, Soundararajan & Bovik 2013 (NIQE)). No-reference metrics are not the only available evidence and do not certify recovered detail. State the degradation, crop, color domain, input count, runtime and sample-selection policy. Chapter 9.3, Image metrics and Chapter 13.12, Measuring a denoiser provide the general evaluation conventions.

14.6.4 The hallucination problem🔗⧉

An adversarially trained method can produce plausible detail that differs from the scene. It can also preserve well-supported structure. The failure to detect is a convincing replacement of an uncertain detail, not sharpness itself.

Faces and text make the consequences concrete. A restored eye shape can alter identity cues; a sharpened glyph can become a different letter. Compare these features with independent evidence when their identity matters.

Three things help and none of them solve it. Show the variability, by drawing several samples where the method permits it, which turns an invisible failure into a visible disagreement. Check consistency, by downsampling the output through the forward model and comparing with the input, which catches results that are not even consistent with the measurement. Concretely: blur and decimate the output with the same kernel and factor the degradation model assumes, subtract the input, and look at the residual. It should be about as large as the input's noise and look like noise; structure in the residual, such as an edge outline or a texture pattern, means the output contains content the input does not support. A small residual is necessary, not sufficient, since many outputs downsample to the same input. And label the output, because a viewer cannot tell by looking which parts were measured.

Part 41, Image Forensics and Authentication considers these consequences. Preserve the input and identify the reconstruction method so that inferred detail is not mistaken for an additional camera measurement.