14.0 SUPER-RESOLUTION AND UPSCALING🔗⧉

A burst can resolve a fine pattern that aliases in one frame because its shifted samples supply different constraints. A single-image method can also improve an enlargement, but must infer unresolved structure from assumptions about images. Modern methods often combine both sources of information.

The important distinction is where the evidence comes from. A fine pattern can fold into the sampled image through aliasing. A fractional camera shift changes that folding, so another frame may separate contributions that the first frame confused. An image prior supplies a different kind of information: which of the compatible fine patterns are common in photographs like the training examples. Both can improve an estimate, and a learned burst method uses both. When judging a result, ask which visible details are supported by additional measurements and which depend mainly on that prior.

14.0.1 What the part inherits, and what it adds🔗⧉

Part 13, Denoising established the role of a noise model and an image prior. Super-resolution adds blur and subsampling to the forward model. Enlarging by four in each dimension creates sixteen output samples per input sample, without creating sixteen independent measurements. Neither the output dimensions nor apparent sharpness establishes how much scene detail was recovered.

14.0.2 The arc of the chapters🔗⧉

Chapter 14.1, What can super-resolution actually do? separates optical attenuation from sampling ambiguity. Chapter 14.2, Multi-frame super-resolution and Chapter 14.3, Learned burst methods use additional captures; the single-image chapters develop interpolation, patch models and learned reconstruction. Chapter 14.6, The perception-distortion tradeoff and perceptual super-resolution explains why low expected error and realistic output statistics can favor different estimates. The remaining chapters address reference images, unknown degradation and video.

14.0.3 One number to carry through🔗⧉

State an enlargement factor per dimension. A $2\times$ enlargement doubles width and height and quadruples pixel count; it need not double resolved detail. Measure the gain on representative scenes, including moving regions where a burst may contribute little beyond one frame.

Contents of this part

• 14.1 What can super-resolution actually do?
  • 14.1.1 What the sensor kept and what it folded
  • 14.1.2 The forward model, and why this is an inverse problem
  • 14.1.3 Two operations, one word
  • 14.1.4 What a resolution number means once a prior is involved
• 14.2 Multi-frame super-resolution
  • 14.2.1 Sub-pixel shifts are extra samples
  • 14.2.2 The classical formulations
  • 14.2.3 Registration is the hard part
  • 14.2.4 What actually ships
  • 14.2.5 The gain, and where it stops
• 14.3 Learned burst methods
  • 14.3.1 Aligning in features, merging by attention
  • 14.3.2 A burst is a set, not a sequence
  • 14.3.3 Where the training data comes from
  • 14.3.4 What a learned merge does that a kernel cannot, and what it costs
• 14.4 Single-image super-resolution: classical methods
  • 14.4.1 Interpolation, and what it actually assumes
  • 14.4.2 Example-based super-resolution
  • 14.4.3 Neighbor embedding and the manifold assumption
  • 14.4.4 Sparse coding and dictionary learning
  • 14.4.5 RAISR: a bank of learned filters, chosen by hashing
  • 14.4.6 The image as its own database
• 14.5 Single-image super-resolution: deep learning
  • 14.5.1 The architectural line
  • 14.5.2 Where the upsampling goes
  • 14.5.3 The ceiling, and what it was made of
  • 14.5.4 What these networks do and do not learn
• 14.6 The perception-distortion tradeoff and perceptual super-resolution
  • 14.6.1 The tradeoff, stated
  • 14.6.2 Buying perceptual quality: perceptual losses and adversaries
  • 14.6.3 Measuring on both axes
  • 14.6.4 The hallucination problem
• 14.7 Generative super-resolution
  • 14.7.1 Conditional generation
  • 14.7.2 Sampling against averaging
  • 14.7.3 Consistency, faithfulness, and what to report
  • 14.7.4 The circle closes
• 14.8 Reference-based and domain-specific super-resolution
  • 14.8.1 Reference-based super-resolution
  • 14.8.2 Face and text restoration
  • 14.8.3 One model, any scale
  • 14.8.4 Learning from the test image alone
  • 14.8.5 The ordering, stated
• 14.9 Blind and real-world super-resolution
  • 14.9.1 The real degradation chain
  • 14.9.2 Degradation modeling: the fix was data
  • 14.9.3 Estimating the degradation instead of covering it
  • 14.9.4 Real paired data, and the gap that remains
• 14.10 Video super-resolution and practical systems
  • 14.10.1 Temporal consistency is the new constraint
  • 14.10.2 Using the neighbors
  • 14.10.3 Hybrid space-time super-resolution
  • 14.10.4 What is deployed
  • 14.10.5 What super-resolution actually delivers
• 14.11 Super-resolution TLDR

▸ full collapsible outline of this part