BookNotesSlidesP-sets Syllabusdraft · v0.2.49.1
◀▲⌂▶
💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.
💡 In a hurry? Jump to this chapter’s 1 big lesson ↓

17.6 HDR+ and burst imaging on phones⧉

The published HDR+ pipeline captures a burst at a common exposure, chosen to protect important highlights, and combines raw frames before demosaicking (Hasinoff et al. 2016). Repeated measurements reduce noise in shadows that look dark in an individual frame. This is a specific capture design, not a description of every phone mode. Night Sight extends burst imaging with motion-aware exposure choices for low light (Liba et al. 2019).

fig-low-light-liba-1
Figure 17.6.1. The same near-dark scene through a prior low-light pipeline versus Night Sight: extending the burst to dozens of frames and adding motion metering yields a clean, sharp, colorful photograph from a fraction of a lux where the earlier method leaves noise and color cast Liba et al. 2019.

Fixed exposure simplifies comparison between frames. Short integration reduces motion blur, while repeated frames collect additional photons. The merge must still handle registration errors, moving subjects and regions with insufficient signal.

These systems connect several parts of the book through a shared engineering investment. HDR+ uses repeated short exposures to improve shadow precision while protecting highlights. Night Sight adapts capture to very low light and motion. Super Res Zoom additionally exploits shifted sampling to resolve fine structure. Alignment, rejection and raw-domain reconstruction recur across them, even though the capture policies and fusion algorithms differ. That common foundation explains why a camera improvement may affect noise, dynamic range and detail together.

17.6.1 Why a phone shoots a burst, and why it underexposes⧉

A handheld camera faces both limited light collection and motion. Longer exposures improve photon statistics but can blur, while shorter exposures require more frames to collect the same total light and pay read noise repeatedly. Capture policy therefore depends on scene motion, illumination and sensor characteristics.

HDR+ uses equal-exposure raw frames to reduce noise while preserving the highlight range of its chosen short exposure. It improves usable shadow range; it does not extend the saturation threshold by measuring brighter highlights at a second exposure. This differs from a conventional bracket, whose frames cover different radiance intervals.

An exposure chosen to protect highlights may leave noisy shadows. Independent repeated measurements can reduce that uncertainty, whereas a highlight clipped in every frame remains only bounded from below. In a normalized noiseless model, protecting a chosen maximum radiance requires

$$ k\,L_{\max} < 1, $$

Noise and uncertain metering require a margin; transient or deliberately unprotected highlights may still clip. Extra frames do not guarantee usable shadows under a fixed time budget. Choose which highlights matter rather than imposing an unconditional no-clipping rule.

💡 Big lesson ((L13.1) · Capture the full set, decide later)

The burst is L13.1 on a phone: capture many identical frames now and defer the noise-versus-range tradeoff to a downstream merge. (L13.1 first appears in this part's introduction; the light-field/plenoptic camera (Part 33, Computational optics and coded imaging) is the same idea for the focus/viewpoint axis, the special case.)

Noise-optimal bracket design (Hasinoff et al. 2010) and HDR+ both account for noise, but they solve different capture problems. A uniform short burst is not the universal optimum of the earlier shutter/ISO optimization. Motion, latency and available buffering also influence the deployed design.

fig-burst-vs-bracket
Figure 17.6.2. Burst, not bracket. Left, classic HDR bracket: a few frames at different exposures (short for highlights, long for shadows); demands a static scene and a steady mount, and the long frame blurs while moving subjects ghost between the differently-exposed frames. Right, HDR+ burst: many frames at one identical, short, deliberately under-exposed setting; hand-held, motion frozen, every frame the same so alignment is easy, and the noise of the short exposure is beaten down $1/\sqrt N$ by merging. The bracket varies exposure across few frames; the burst holds exposure fixed across many.
fig-why-underexpose
Figure 17.6.3. The irreversibility asymmetry. Top, a single "correctly" exposed frame: the bright window clips to flat white; the detail above the full-well ceiling is destroyed and unrecoverable. Bottom, an under-exposed burst: the same window stays below clipping in every frame, so its detail survives, while the now-darker, noisier shadows are cleaned up $1/\sqrt N$ after the merge. A clipped highlight is gone forever; a noisy shadow is only noisy, so bias the exposure down ($k\,L_{\max}<1$) and recover the shadows by computation.

The paper's own teaser shows what the underexposed burst buys at the hardest extreme: a near-dark, hand-held portrait.

fig-hdrplus-1
Figure 17.6.4. HDR+ in the dark. A near-black, hand-held selfie that a single short frame renders as noisy murk (left) becomes a clean, well-exposed portrait (right) after a burst of identical underexposed raws is aligned and merged; the eye crops show grain beaten down and detail restored, with no motion blur. The whole multiple-exposure part, shipped to a phone (Hasinoff et al. 2016).

Compare a shadow patch with a lamp that is saturated in every frame. The shadow measurements can vary around a useful underlying signal, so combining independent observations can make that signal more precise. The clipped lamp values do not vary with how far the true signal exceeded the ceiling; averaging the ceiling repeatedly supplies no measurement of that excess. This asymmetry motivates reserving highlight headroom and paying some of the cost as shadow noise. It does not make arbitrarily short exposures desirable: read noise, total capture time, rejected frames, and values lost below the usable floor still constrain what the burst can recover.

17.6.2 The HDR+ pipeline: align and robust-merge in raw⧉

Choose a reference, estimate local alignment and combine consistent raw samples. Maintain CFA phase, black-level and gain conventions. After raw merging, demosaic and render the result. The exact interpolation and alignment precision depend on the implementation; the original HDR+ and later burst-super-resolution pipelines should not be conflated.

Raw merging retains access to sensor-domain measurements and a calibrated noise model. Interpolation and merging can still introduce correlations. Joint reconstruction or a suitably modeled RGB pipeline are alternatives; pre-demosaic processing is a design choice rather than a universal optimality result.

A simplified spatial-weighted model illustrates exposure normalization, although the actual HDR+ merge operates on tile transforms:

$$ \hat L(x,y) \;=\; \frac{\sum_i w_i\,I_i\big(W_i^{-1}x\big)}{k\sum_i w_i}, $$

HDR+ uses a transform-domain merge on aligned tiles, with shrinkage informed by estimated noise and differences from the reference. This differs from simply assigning a per-pixel inverse-variance weight. Large inconsistencies reduce temporal averaging; a reference fallback sacrifices noise reduction to limit motion artifacts. Equal averaging provides a useful independent-noise reference case:

$$ \sigma_{\text{merged}} \;\approx\; \frac{\sigma_{\text{single}}}{\sqrt N}, $$

This approximation assumes equal noise and independent, correctly registered samples. Rejection, adaptive filtering and correlated errors change it.

Local alignment handles some motion that a global transform misses. Occlusions and varying exposure blur remain difficult, so the merge needs confidence and fallback even after alignment. For a merged image, inspect both detail and the spatial variation in noise rather than reporting burst length as the number of useful samples everywhere.

fig-hdrplus-pipeline
Figure 17.6.5. The HDR+ flow. Left to right: a burst of $N$ identical under-exposed raws → pick a sharp reference frame → align every other frame to it (tile-based, sub-pixel) → robust merge in raw (Wiener-style; reject mis-aligned or moving tiles) → a single denoised, high-DR raw → demosaic → tone-map and finish → JPEG. The decisive choice is that the merge, which does denoising and HDR together, happens in raw, before demosaicking.

The original paper draws the same flow on a real scene, and shows both why alignment is decisive and how the finished pipeline compares to a conventional ISP.

fig-hdrplus-3
Figure 17.6.6. The HDR+ pipeline as the authors drew it. From a live viewfinder fed by the hardware ISP, the shutter press grabs a burst of raw frames at one chosen exposure-and-gain; these are aligned and merged at full resolution, then run through white balance, demosaic and chroma denoise, a local tone map (exposure fusion Mertens et al. (exposure fusion)), dehaze and global tone map, and a final sharpen and saturation pass. The decisive step, align-and-merge, sits before demosaicking, so denoising and HDR happen together on the raw signal (Hasinoff et al. 2016).
fig-hdrplus-6
Figure 17.6.7. Why alignment decides the result, on a night street. The same moving scene merged four ways: a single reference frame (noisy), a naive temporal mean (motion smears the sign and the passing vehicle into a ghost), the mean with alignment (the static sign sharpens, but residual motion still ghosts), and the robust merge with alignment (the sign is crisp and the moving vehicle is handled without a double image). Alignment plus the outlier-rejecting weight is exactly what turns a helpful average into a clean result (Hasinoff et al. 2016).
fig-hdrplus-2
Figure 17.6.8. Three ways a burst merge goes wrong, named. (a) Ghosting, a moving subject left a double image where the merge trusted misaligned frames; (b) excessive denoising, over-aggressive averaging smeared away real texture into a waxy blur; (c) excessive range compression, the tone map pushed local contrast flat and hazy. The failure gallery that the robust weight, a light hand on denoising, and a restrained tone curve each exist to avoid (Hasinoff et al. 2016).
fig-hdrplus-10
Figure 17.6.9. Conventional ISP versus HDR+, across hard scenes. A dim church interior, a low-light living room, and a moving child are each shown as a conventional single-frame ISP result beside the HDR+ result (and, for the child, the cost of not merging). The burst pipeline holds highlights, cleans shadows, and freezes motion where the conventional pipeline leaves noise, blur, or clipping (Hasinoff et al. 2016).

Tile alignment makes a local motion approximation. A tile on a stationary wall and a tile on a moving arm can choose different displacements, whereas one global translation must compromise between them. But a tile straddling the arm boundary contains two motions. No single displacement aligns both, and a newly revealed background patch has no corresponding foreground sample to average. The robust merge handles some of these failures by reducing inconsistent contributions. Where fewer observations survive, more noise remains. The alignment and merge therefore need to be understood together: locally different noise levels can be the visible consequence of preserving moving structure.

17.6.3 From burst HDR to burst super-resolution⧉

Wronski and colleagues reconstruct color from a shifted raw burst using a fusion model designed for nonuniform sampling (Wronski et al. 2019). Burst super-resolution requires more than changing output size: it needs accurate sampling geometry, usable subpixel phases and a reconstruction rule matched to those samples.

Hand motion can place scene samples at complementary subpixel locations. Accumulating them on a finer grid can improve resolution when the optics retain useful high-frequency information. Repeated or inaccurate offsets provide less benefit, and independently moving content may require a reference fallback (Chapter 14.2, Multi-frame super-resolution).

Different offsets give aliased components different phases, supplying additional constraints. Recovery depends on rank, optical transfer and noise; image priors still influence the answer. Raw burst reconstruction can jointly perform color reconstruction, avoiding a separate single-frame demosaicking stage (Chapter 14.3, Learned burst methods).

fig-super-res-zoom-1
Figure 14.2.5 (reproduced). Super Res Zoom on a phone. A handheld burst, aligned to sub-pixel precision and fused onto a finer grid, recovers real detail well past what a single frame (or a digital crop of it) can resolve, the feature Google shipped as the Pixel's "Super Res Zoom." Reproduced from Wronski et al. (2019) with permission.
fig-super-res-zoom-9
Figure 17.6.10. Super Res Zoom, robust to motion. A handheld burst of a street scene with a bus moving through it. Left: merging the aligned frames with no robustness test ghosts the moving bus. Middle: the robustness map, dark where local motion makes a frame disagree with the reference, so those samples are down-weighted. Right: the robust merge keeps the bus whole while still fusing the static background. Reproduced from Wronski et al. (2019) with permission.
Video 17.6.1. Video: Handheld Multi-Frame Super-Resolution (Wronski, Garcia-Dorado, Ernst, Kelly, Krainin, Liang, Levoy and Milanfar 2019) (YouTube).

HDR merging, burst denoising and burst super-resolution share registration and combination machinery. Their forward models and capture objectives differ. Preserve those distinctions when comparing quality, capture time and processing cost.

Recap: big lessons of this chapter

💡 Big lesson ((L13.1) · Capture the full set, decide later)

The burst is L13.1 on a phone: capture many identical frames now and defer the noise-versus-range tradeoff to a downstream merge. (L13.1 first appears in this part's introduction; the light-field/plenoptic camera (Part 33, Computational optics and coded imaging) is the same idea for the focus/viewpoint axis, the special case.)