BookNotesSlidesP-sets Syllabusdraft · v0.2.49.1
◀▲⌂▶
💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.

14.5 Single-image super-resolution: deep learning⧉

A learned upscaler maps a small image to a larger one using regularities in training pairs. SRCNN (Dong et al. 2014 (SRCNN)) organized this mapping into patch-feature extraction, nonlinear mapping and reconstruction after a bicubic enlargement.

Those stages resemble a patch-based pipeline but are trained jointly. The comparison is architectural; SRCNN is not exactly sparse coding with its variables renamed.

Subsequent designs changed depth, residual paths, feature mixing and the stage at which spatial enlargement occurs.

14.5.1 The architectural line⧉

VDSR (Kim, Lee & Lee 2016 (VDSR)) made the network deep and predicted the residual: the difference between the bicubic upsample and the truth, rather than the image. The argument is the one DnCNN Zhang et al. 2017 (DnCNN) makes in Chapter 13.7, Learned denoisers. The input and the output are nearly the same image, so a network that outputs the difference does not spend its capacity learning to copy, and the target often has smaller magnitude. The residual can still contain low frequencies; their absence is not guaranteed. Depth then became useful rather than merely expensive, and one network could handle several scale factors.

LapSRN (Lai et al. 2017 (LapSRN)) upsampled progressively, in factors of two through a Laplacian-pyramid structure, with a loss at every level. Going straight to eight times in one step asks the network to invent a great deal at once; going in stages lets each level work on a problem that resembles the one the previous level solved.

EDSR (Lim et al. 2017 (EDSR)) removed batch normalization from its residual blocks and used the saved memory for a larger model. Their argument was that batch normalization rescales features by statistics of the training batch, which constrains the range of values a restoration network can carry through its layers, while removing it also saved roughly 40% of training memory. The paper reports this as an effective restoration design; batch normalization does not literally erase all absolute brightness information, since learned affine parameters and other paths also matter.

RCAN (Zhang et al. 2018 (RCAN)) added channel attention to residual groups. SwinIR (Liang et al. 2021) combines convolution with shifted-window transformer blocks. These are representative design choices, not evidence that one architecture is universally most accurate or efficient.

The location of upsampling has a direct effect on the cost of these feature operations.

Channel attention changes which learned features contribute for the current input. A network can emphasize channels useful for a strongly oriented edge and reduce others, instead of applying the same feature weighting to every image. Windowed attention makes a different trade: restricting interactions to windows bounds their cost, while shifting the windows in successive blocks lets information cross the previous boundaries. These choices explain how RCAN and SwinIR use context; they are more informative than comparing depth or parameter count alone.

14.5.2 Where the upsampling goes⧉

SRCNN upsampled the input with bicubic first and then processed at full resolution. Every convolution therefore runs on the enlarged grid, increasing feature-map storage and convolution cost.

Many designs process features at low resolution and enlarge near the output using a sub-pixel convolution (Shi et al. 2016 (sub-pixel convolution)). A convolution produces $Cr^2$ channels, then pixel shuffle rearranges them into $C$ channels at $r$ times the width and height. The rearrangement adds no samples or arithmetic inference by itself. Keeping a comparable trunk at low resolution reduces its spatial work by roughly $r^2$; whole-network savings depend on channel counts and the output head. Relations to transposed convolution require compatible kernel layouts and parameterization.

Algorithm 14.5.1
# sub-pixel convolution: learn at low resolution, rearrange at the end
feat = trunk(y)                       # all the depth runs at LOW resolution
feat = conv(feat, out_channels = C * r * r)
x    = pixel_shuffle(feat, r)         # (C*r*r, H, W) -> (C, H*r, W*r)
x    = x + upsample_bicubic(y)        # global residual: learn the detail, not the image

Different output phases are produced by different feature channels, which can create periodic artifacts if their learned responses disagree. Pixel shuffle is only a rearrangement; such artifacts arise from the predictions. Fine spatial relationships can be represented in channels before shuffling, so low-resolution processing does not impose a categorical ban on reasoning about fine structure.

fig-sr-architecture-anatomy
Figure 14.5.1. Interactive. One super-resolution pipeline drawn as a network, with its design choices switchable, but built from fixed Gaussian filters rather than learned ones, so it shows where the cost of each choice falls, not what training buys. Residual and direct prediction give the same output here; the residual view shows what a residual network would be asked to predict, which is mostly edges. Upsampling at the start versus at the end changes the cost sharply: processing at low resolution costs the square of the scale factor less per output pixel, because every layer runs on that many fewer pixels. The normalization switch only rescales a gain, and the late upsampler is bicubic interpolation, not a learned sub-pixel layer. Cost counts cover the blurs alone; there are no learned parameters to count. Interactive: toggle each design choice, read the cost per output pixel, and compare the output with the residual it implies.

14.5.3 The ceiling, and what it was made of⧉

DIV2K Agustsson & Timofte 2017 (DIV2K) provides high-resolution images and specified degradation tracks; Set5 Bevilacqua et al. 2012 (Set5), Set14 Zeyde, Elad & Protter 2010, BSD100 Martin et al. 2001 (Berkeley segmentation dataset) and Urban100 Huang, Singh & Ahuja 2015 (Urban100) commonly supply evaluation images. A benchmark result must identify how its low-resolution inputs were made, its scale, border crop and color scoring domain. The image collection alone does not specify bicubic downsampling.

Small improvements on a fixed benchmark can reflect data, model, optimization and evaluation limits. A flattening empirical curve does not prove that the Bayes-optimal error has been reached.

Under population squared-error training with an unrestricted estimator, the optimum is the conditional mean. An $L_1$ loss instead selects componentwise conditional medians. Finite networks approximate their training objectives, and uncertain texture can become smooth under either point-estimation approach. EDSR and other restoration systems need their actual loss identified before assigning a posterior statistic to them.

Chapter 14.6, The perception-distortion tradeoff and perceptual super-resolution explains a statistical tension between expected fidelity and output realism. It does not establish that a particular architecture or benchmark curve has reached that theoretical limit.

fig-sr-psnr-ceiling
Figure 14.5.2. Interactive. The ceiling, reproduced on a toy. The fixed filter stack of the previous figure is measured at increasing depth, with its distortion score and its cost reported at each, and the two curves scaled independently so their shapes can be compared. The score flattens while the cost keeps climbing, the same shape as the published curve of PSNR against publication year for learned methods; this is an analog of that curve on a hand-built pipeline, not the benchmark curve itself, and it contains no learned methods and no perceptual scores. Interactive: step through the depths and read the score and the cost at each.

14.5.4 What these networks do and do not learn⧉

Training chooses the input degradation and the target statistic. Test a model on the blur, noise, compression and image types it will encounter; bicubic benchmark performance does not establish real-camera performance. Pixel losses do not guarantee conservative failures: a deterministic network can alter text or create repeatable false patterns. Inspect those cases alongside average error.