14.8 Reference-based and domain-specific super-resolution🔗⧉

A photograph of a building may have a sharper reference in another view. A face or a printed page may instead benefit from a model trained for that class of image. Extra reference data and a more specialized prior help in different ways.

A reference supplies measurements of another view or time. A specialized model supplies assumptions fitted to its training population. Neither guarantees a better reconstruction: reference mismatch and class mismatch can both introduce errors.

Evaluate whether the added information applies to the particular region being restored.

14.8.1 Reference-based super-resolution🔗⧉

The setup is common, and how common it is matters. A phone has a wide and a telephoto camera, so a wide shot has a high-resolution reference for part of its field. A photographer has an older, closer shot of the same building. A tourist photograph of a landmark has ten thousand references online. In each case there exists a picture with real detail of the same subject, at a different scale, from a different viewpoint, under different light.

Reference-based methods transfer that detail. The reference is matched to the input, densely and at the feature level with geometric alignment when available, and appearance matching to handle remaining differences; matching patches are swapped or attended over; and a network fuses the transferred texture with the input's own structure. SRNTT (Zhang et al. 2019 (SRNTT, reference-based SR)) does it with a patch-based neural texture transfer, TTSR (Yang et al. 2020 (TTSR)) with a transformer whose attention is the matching step.

Read against Chapter 14.2, Multi-frame super-resolution, this is the same idea with the constraints relaxed: another observation of the same scene, not aligned, not simultaneous, not from the same camera. That relaxation is what makes it useful and what makes it fail. The failures are specific: where the reference does not contain the region, the method should reduce transfer and rely on its input-only path; where the matching is confidently wrong, texture from the wrong part of the reference arrives sharp and misplaced; and where the lighting differs, transferred texture carries the reference's shading.

The fair reading is that the output is a blend of measurement and invention whose proportions vary across the frame, which is a good reason for a method to report where its transfers came from.

14.8.2 Face and text restoration🔗⧉

Narrow the class to faces and the prior becomes extraordinarily strong, because faces share a structure that almost nothing else does: two eyes, a nose, a mouth, in nearly the same arrangement, at nearly the same relative sizes, with skin between them.

GFPGAN uses a pretrained face generator as a prior (Wang et al. 2021 (GFPGAN)). CodeFormer predicts entries in a learned discrete latent codebook and decodes them into a restored face (Zhou et al. 2022 (CodeFormer)). These codes are learned features, not a catalog of literal eyes and mouths. Model, fidelity control and degradation determine the balance between preserved input structure and prior-driven changes.

Both approaches can change identity-relevant features when the input is ambiguous. Their outputs can also be accurate on supported detail. Compare against independent photographs when identity matters; a realistic face alone cannot establish which fine features were present.

Standard deterministic inference need not change when run twice. Only a stochastic model or explicit perturbation experiment supplies variation across runs; repeated identical output is not evidence of certainty.

Text is the same story with a legible failure. A model with a glyph prior turns an unreadable sign into crisp letters, and the letters can be the wrong ones. It is the best teaching example in the part precisely because a reader can see the error, where in a face they cannot.

A family photograph makes the issue less abstract. If nobody remembers the person's fine facial features, a convincing restoration may become the version the family remembers. A reconstruction that changes an eyelid or mouth shape can therefore matter even when no identification decision is involved. Preserve the original alongside the enhanced version so that the model's interpretation does not replace the surviving record.

14.8.3 One model, any scale🔗⧉

Pixel-shuffle models commonly fix integer enlargement factors. Classical resampling already supports arbitrary coordinates; learned implicit representations extend this flexibility to learned reconstruction.

LIIF predicts color at a continuous coordinate from local image features (Chen, Liu & Wang 2021 (LIIF)); Meta-SR predicts reconstruction weights conditioned on scale (Hu et al. 2019 (Meta-SR)). For example, write $\hat x(q)=f_\theta(F(y),q,c)$, with feature map $F(y)$, query coordinate $q$, and output-pixel footprint $c$. The footprint distinguishes a pixel average from a point evaluation. Arbitrary query spacing does not guarantee correct detail at untested scales or antialiased reduction.

The same shift shows up in the rendering and 3-D parts of this book, where an implicit function replaces a sampled grid for the same reason.

The practical request is often a print size or a crop, not an integer enlargement. A requested width may require a factor of 2.3 rather than two or four. A coordinate-conditioned model can evaluate the requested grid directly instead of generating a fixed larger grid and resampling it again. The output-pixel footprint still matters because changing the grid changes the area each sample should represent.

14.8.4 Learning from the test image alone🔗⧉

The last narrowing is the extreme one: train on the single image you are trying to upscale.

ZSSR (Shocher, Cohen & Irani 2018 (ZSSR)) takes the input, downscales it to make low-and-high pairs from the image's own content, trains a small network on those pairs at test time, and applies it to the input. There is no external dataset anywhere in the method. The authors demonstrate it on images far from typical training sets, such as old photographs, aerial views, microscopy and images with repeated text or architectural patterns, where recurrence within the image is strong and an external dataset's degradation is a poor match. It is Chapter 14.4, Single-image super-resolution: classical methods's cross-scale recurrence turned into a training set rather than a patch database.

Internal learning can adapt to image-specific recurrence and an estimated degradation. It is not immune to model mismatch: the extra downsampling used to make training pairs must resemble the relevant degradation, and noise or compression can be learned as texture. Unique structures may supply few useful internal examples.

Training per image adds latency. Evaluate the adaptation against a fixed pretrained model and an interpolation baseline on the intended image type.

14.8.5 The ordering, stated🔗⧉

There is no universal order of fidelity or sharpness among these approaches. A valid reference can add subject-specific measurements; an incorrect one can transplant detail. Internal recurrence can fit unusual imagery but may be weak. Class priors can help within their domain and alter unusual features. Choose using representative tests, reference consistency and the importance of preserving identity or text.