14.3 Learned burst methods🔗⧉

A burst supplies repeated and shifted measurements, but their value depends on blur, noise, motion and visibility. A learned reconstruction can use local context to align frames and decide how much each contributes.

Learning can estimate correspondence, combine aligned features, or fuse frames without an explicit registration stage. These choices have different assumptions; none removes the sampling limits of Chapter 14.2, Multi-frame super-resolution.

14.3.1 Aligning in features, merging by attention🔗⧉

Deep Burst Super-Resolution (Bhat et al. 2021) extracts features from raw frames, uses optical-flow-based alignment, and combines aligned features through learned attention. Feature extraction provides context; it does not guarantee that features are less noisy or correspondence is correct.

Learned fusion can use context to distinguish noise, motion and occlusion. Its weights remain predictions, and a network can accept an incorrect match. Inspect fusion confidence and compare with a single-frame result where neighboring frames disagree.

Bhat and colleagues' separate Deep Reparametrization of Multi-Frame Super-Resolution and Denoising paper moves a MAP reconstruction into a learned latent space, using a learned error metric and an unrolled optimization (ICCV 2021 paper). This connects burst fusion to Chapter 13.7, Learned denoisers: retain a reconstruction problem and its iterative structure, while learning how images and errors are represented. It differs from the feed-forward attention fusion of Deep Burst Super-Resolution.

14.3.2 A burst is a set, not a sequence🔗⧉

Frame-order invariance and alignment are separate design choices.

For a static scene and a fixed output coordinate system, permuting the nonreference frames should not change the result. Reference-frame identity and per-frame metadata must remain attached to their images. Dynamic scenes may require timestamps or a specified target time. A symmetric architecture can impose order invariance without proving that the frames are geometrically aligned.

Aittala and Durand (Aittala & Durand 2018 (permutation-invariant burst deblurring)) build the invariance into the architecture instead. Each frame passes through the same per-frame encoder; at intervals, the frames exchange information through a symmetric pooling operation, an average or a maximum across the frame axis, which by construction does not care about order; each frame then continues with both its own features and the pooled summary. Formally the network computes

with $f$ shared across frames and the pooling symmetric, so any permutation of the inputs gives the same output. The number of frames becomes a runtime parameter rather than an architectural one, and the architecture can accept different positive frame counts. Quality at counts outside training still needs evaluation.

# schematic symmetric fusion; assumes a nonempty burst in a common coordinate convention
feats = [encode(y) for y in burst]              # SAME weights for every frame
for layer in range(L):
    shared = mean(feats)                        # symmetric: order cannot matter
    feats  = [block(cat(f, shared)) for f in feats]   # each frame sees itself and the consensus
xhat = decode(mean(feats))
# frames may arrive in any order, and K may differ from training

Aittala and Durand demonstrate burst deblurring, where different exposure trajectories give frames different blur (Aittala & Durand 2018 (permutation-invariant burst deblurring)). A displacement field alone cannot remove that difference, although registration can still align gross scene position. Their network learns to fuse the differently degraded inputs without an explicit alignment module; pooling alone is not a general solution to arbitrary camera motion.

Use symmetric aggregation when the target is independent of input order. For a video target tied to a particular time, preserve the temporal information the reconstruction needs.

Different blur kernels complicate the correspondence problem. Suppose one exposure has predominantly horizontal camera shake and another predominantly vertical shake. Translating one image can align the gross scene position, but cannot turn its horizontal streaks into vertical ones. Nevertheless, the two measurements can complement each other: a spatial frequency strongly attenuated by one kernel may survive the other. A burst method can combine that evidence without requiring any single input frame to be sharp. This is the motivation for learning a fusion of differently degraded observations; treating the burst as a set also avoids assigning significance to its arbitrary input order.

14.3.3 Where the training data comes from🔗⧉

A low-resolution burst and a sharp high-resolution reference must depict the same scene, but their cameras can differ in geometry, color and optics. Synthetic training gives exact correspondence under a chosen simulator; captured pairs test effects the simulator may miss.

BurstSR uses handheld Samsung Galaxy S8 raw bursts and Canon 5D Mark IV DSLR references, with geometric and photometric alignment (Bhat et al. 2021). It should not be described as simultaneous beam-splitter capture. Changes between captures and imperfect alignment limit how closely the reference represents the burst.

14.3.4 What a learned merge does that a kernel cannot, and what it costs🔗⧉

A learned merge can exploit image context beyond a local regression rule, but its behavior depends on training conditions. Compare it with an adaptive classical merge on matched inputs, including blurred frames, occlusions and out-of-distribution content. Both methods can lose detail or introduce artifacts. A burst constrains the result more strongly only where its extra frames carry useful, correctly associated measurements.