14.11 Super-resolution TLDR⧉
- Capture useful detail first. Better focus, adequate exposure and a closer view can supply evidence an upscaler lacks. Preserve raw bursts when available; extra frames help only where their sampling, motion and noise are usable (Chapter 14.2, Multi-frame super-resolution).
- Treat the enlargement factor as output dimensions. Four times per dimension means sixteen times the pixels, not sixteen times the information. Compare resolved features at the intended display or print size.
- For one processed image, compare interpolation with a model trained for similar blur, noise and compression. Inspect text, faces and repeating patterns; a sharper result can contain incorrect detail (Chapter 14.9, Blind and real-world super-resolution).
- Use generative enlargement when plausible appearance suits the task. Preserve the input and identify the processing. Sample variation shows model alternatives; agreement does not establish truth (Chapter 14.7, Generative super-resolution).
- For video, inspect flicker, ghosting, scene cuts and latency. Temporal fusion can help, but unreliable history should be rejected rather than carried forward (Chapter 14.10, Video super-resolution and practical systems).
- Evaluate on representative images. Report reference error and appearance separately, check agreement with the assumed measurement model, and measure cost. Neither a benchmark rank nor a no-reference quality score certifies recovered detail.
Separate information supported by the measurements from information supplied by the image model. Subpixel-shifted frames can sample scene detail at additional positions, provided blur, motion, and noise leave that detail usable. A single-image method relies more heavily on image regularities to choose among high-resolution explanations. This is a difference in available evidence, not a guarantee that every burst result is correct or every single-image estimate is arbitrary (Chapter 14.1, What can super-resolution actually do?).
The forward model describes blur, sampling, and noise in their actual order. A burst adds a different warp or sampling phase for each frame. Useful shifts can separate aliases, but repeated identical samples mainly improve noise. Frequencies suppressed by all observations remain unconstrained by those observations; a prior can suggest them without making them measured detail. Registration errors and moving objects can erase the benefit of additional frames (Chapter 14.2, Multi-frame super-resolution).
Choose the method family by the evidence and the intended use:
| Family | What it uses | What to inspect |
|---|---|---|
| Interpolation | Nearby measured samples and a reconstruction kernel | Softness, ringing, aliasing; extra pixels do not establish extra resolved detail |
| Multi-frame reconstruction | Complementary samples and repeated measurements | Alignment, occlusion, blur differences, ghosting and actual detail gain |
| Classical single-image methods | Local structure, patch dictionaries, or internal recurrence | Whether the assumed structure fits this image |
| Squared-error-trained networks | The training distribution of sharp/degraded pairs | Averaged-away texture and mismatch to the real degradation |
| Adversarial or generative methods | A learned preference or distribution for plausible detail | Incorrect texture, identity changes, and variation between possible outputs |
The loss helps explain the appearance. Under squared error, the ideal estimator is the conditional mean. When several sharp explanations remain plausible, averaging them can soften uncertain structure. A perceptual or generative objective can select sharper-looking alternatives, at a different reference error. The perception-distortion tradeoff concerns distributions and an estimation objective; it does not establish that every sharper result is better, or that architectures and training data no longer matter (Chapter 14.6, The perception-distortion tradeoff and perceptual super-resolution).
A narrow prior is useful and dangerous for the same reason: it knows which details are typical. A face model may produce a plausible feature that is wrong for the person photographed. A reference image adds real observations of a subject, but transferring its detail still requires correspondence and agreement about pose, lighting, and scene state. Internal learning exploits recurrence within the input and can still fail when the degradation model is wrong. Continuous-coordinate representations allow a requested scale without requiring a separately trained network for each integer factor (Chapter 14.8, Reference-based and domain-specific super-resolution).
Real photographs are rarely clean bicubic downsampling of a sharp reference. They have passed through optics, noise, demosaicking, sharpening, compression, and possibly several resizes. Test with representative degradations and cameras. A benchmark gain under one synthetic operator can disappear on another (Chapter 14.9, Blind and real-world super-resolution). At modest computational budgets, a learned filter bank such as RAISR Romano, Isidoro & Milanfar 2016 (RAISR) is also a useful point of comparison: local gradient measurements select a filter without running a large network (Chapter 14.4, Single-image super-resolution: classical methods).
For evaluation, report fidelity and appearance separately, alongside agreement with the assumed forward model and computational cost. Multiple generated samples reveal alternatives the model considers plausible, but their spread need not be calibrated uncertainty. For video, evaluate sequences: individually attractive frames can shimmer when their invented detail changes from frame to frame. Temporal methods must also reject unreliable history at occlusions and scene cuts.