14.9 Blind and real-world super-resolution🔗⧉

An upscaler trained on clean bicubic reductions may sharpen JPEG blocks or sensor noise in a camera image. Test this by applying the same model to a controlled reduction and to a processed photograph containing those artifacts.

The training degradation specifies the inverse problem the network learns. Mismatch can reduce accuracy, although its size and direction depend on the model. Bicubic reduction remains a useful controlled benchmark; it is not a complete camera simulator.

14.9.1 The real degradation chain🔗⧉

A processed photograph can include optical and pixel-aperture blur, CFA sampling, signal-dependent noise, demosaicking, sharpening, nonlinear rendering, compression and repeated resizing. Exposure and calibrated sensor parameters determine the noise; ISO alone does not specify it. A simplified degradation model retains only blur, resampling, noise and compression:

Here $k$ is an unknown blur, $r$ a scale factor, $n$ noise and $\mathcal C$ compression. This equation omits the CFA and most rendering stages. Clean bicubic reduction specifies a resampling kernel without these additional degradations; training on it does not teach a model how to distinguish all camera artifacts from scene detail.

Blur, noise, nonlinear processing and resizing each change the inference problem. Inspect their contributions separately rather than attributing every artifact to one mismatched resize kernel.

Follow a photograph through two resizes. The camera first blurs and samples the optical image, reconstructs color, reduces noise, sharpens, and encodes it. An editor then downsizes the result and saves a JPEG; a sharing service may decode, resize, and compress it again. The second resize acts on ringing and block errors already introduced by earlier stages. It cannot generally be represented by applying one clean blur kernel to a pristine high-resolution photograph and then adding independent noise. Synthetic training pairs must therefore represent relevant orders and repetitions of operations, as well as their parameter ranges. A visually similar low-resolution input can have a different relationship to its target when that history changes.

14.9.2 Degradation modeling: the fix was data🔗⧉

If the problem is that training data does not resemble reality, generate training data that does. Take high-resolution photographs and degrade them yourself, sampling the whole chain at random, then train on those pairs.

BSRGAN (Zhang et al. 2021 (BSRGAN)) samples a random degradation: a blur kernel drawn from a family, a downsampling method drawn from several, noise of a random type and level, and JPEG compression at a random quality, in a shuffled order. The shuffling matters more than it sounds, because a real image may have been compressed before it was resized or after, and a model trained on one order does not handle the other.

Real-ESRGAN (Wang et al. 2021) uses a high-order pipeline, applying the whole degradation chain twice, on the argument that images on the internet have usually been through more than one processing pass. It also models the ringing and overshoot that sharpening produces, which simple blur-and-resize models omit.

# a randomized degradation pipeline, the essential shape
def degrade(x, target_size):
    for _ in range(random_order_passes()):        # once for camera, again for the web
        k = sample_blur_kernel()                  # isotropic, anisotropic, motion
        x = convolve(x, k)
        x = resize(x, sampled_intermediate_size(), method=sample_resize_method())
        x = x + sample_noise(x)                   # Gaussian, Poisson, or processed-sensor
        x = jpeg(x, quality=uniform(30, 95))
    return resize_to_target(x, target_size)
# sample fresh parameters; keep the final output size equal to the requested scale

Training on a broader degradation family can improve performance on real processed images while changing scores on a specific synthetic track. The direction of that change must be measured. BSRGAN and Real-ESRGAN address important mismatches without covering every camera, compression history or subject.

14.9.3 Estimating the degradation instead of covering it🔗⧉

An alternative to broad augmentation is to estimate degradation parameters and condition the restoration on them. This resembles the noise-level conditioning of Chapter 13.7, Learned denoisers, but includes blur and sampling.

Iterative kernel correction (Gu et al. 2019 (iterative kernel correction)) alternates between estimating the blur kernel and restoring the image, using the artifacts in the current restoration as evidence about how wrong the kernel is: restoration artifacts help the learned correction, but do not uniquely diagnose kernel width. KernelGAN (Bell-Kligler, Shocher & Irani 2019 (KernelGAN)) estimates the kernel from the input alone, by training a small generator so that the downscaled image keeps the same patch distribution across scales, which is the cross-scale self-similarity of Chapter 14.4, Single-image super-resolution: classical methods used as a signal rather than as a source of patches.

Both are the blind-deblurring problem of Chapter 16.4, Blind deblurring wearing different clothes, and both inherit its difficulty: image and kernel are being estimated together, the data does not determine them separately, and a prior has to break the tie.

14.9.4 Real paired data, and the gap that remains🔗⧉

The third route is to capture real pairs. Photograph the same scene twice with different focal lengths, so the long shot is the high-resolution reference for the wide one, then align and calibrate them. RealSR (Cai et al. 2019 (RealSR)) and its relatives are built this way, and provide useful captured references.

Captured pairs retain uncertainty from alignment, camera response, focus and scene changes. RealSR includes multiple cameras, but any captured collection still samples a limited set of optics and processing conditions. Changing focal length from the same camera center changes magnification, not perspective by itself; camera displacement and lens distortion complicate registration. Different defocus cannot generally be removed by a geometric warp.

Specify the assumed degradation and test the model on representative camera images. Captured and synthetic benchmarks answer different questions; report their reference construction and processing history.