BookNotesSlidesP-sets Syllabusdraft · v0.2.49.1
◀▲⌂▶
💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.

17.3 Merging exposures⧉

Calibration puts the exposures into common units. The next decision is how much to trust each estimate at each location. We start with a simple method based on thresholds. But we then observe that long exposures can supply precise shadows and clipped highlights at the same time, so a more advanced merge must combine both measurement values and their reliability.

17.3.1 The weighted merge⧉

We work with registered, black-level-corrected linear values $I_i$ and calibrated exposure factors $k_i$. We divide by $k_i$ to put the observations in common relative-radiance units, then combine the valid estimates. For encoded inputs, we first invert the appropriate response.

Validity masks reject saturated, defective or unregistered samples. Noise weights reduce the influence of uncertain measurements. We choose clipping margins in the input's calibrated units; thresholds such as 0.99 or 0.002 are not universal. Very dark samples can still provide information when their noise is modeled. We avoid selecting on noisy intensity more aggressively than necessary, since this can bias the merge.

We let $w_i$ include both validity and reliability. The linear-light merge is

$$\hat L(x,y)=\frac{\sum_iw_i(x,y)I_i(x,y)/k_i}{\sum_iw_i(x,y)}.$$

In words, the estimated radiance is the reliability-weighted mean of the rescaled observations. In code this is one pass per pixel over the bracket: divide out each exposure factor, accumulate by reliability, normalize:

Algorithm 17.3.1
for each pixel p:
    num, den = 0, 0
    for each exposure i:
        if valid(i, p):
            q = linear_value(i, p) / exposure_factor[i]
            w = radiance_weight(i, p)
            num += w * q
            den += w
    if den > 0:
        L[p], measured[p] = num / den, True
    else:
        L[p], measured[p] = flagged_fallback(p), False

For a calibrated nonlinear response, Debevec–Malik Debevec & Malik 1997 instead combines log-radiance estimates:

$$ \ln \hat L(x,y) = \frac{\sum_i w(I_i)\,\big(g(I_i) - \ln t_i\big)}{\sum_i w(I_i)}, $$

Exponentiation produces a weighted geometric mean, not the preceding arithmetic mean. The error model and weighting should match the domain. Multiple independent, equally precise observations reduce standard deviation by $1/\sqrt N$; unequal weights, correlations and data-dependent selection require the more general variance calculation (Chapter 13.2, Denoising by averaging).

17.3.2 When every exposure is bad at a pixel⧉

If every exposure clips, the shortest exposure supplies the strongest lower bound on radiance. If all are too noisy, the longest usable exposure is usually the best fallback at fixed gain, provided motion and defects do not invalidate it. We keep an uncertainty or validity flag, since neither case supplies a reliable point estimate. With variable gain, we select by calibrated radiance-domain uncertainty rather than by exposure duration alone.

Chapter 18.4, Exposure fusion directly blends rendered inputs using appearance weights. It is useful when a finished picture is the objective; its output is not a calibrated radiance measurement.

fig-weight-triangle
Figure 17.3.1. The reliability weight $w(I)$ as a function of pixel value. The weight is zero at the clipped end (e.g. $I \ge 0.99$) and zero at the dark, noisy end (e.g. $I \le 0.002$), rising to a plateau in the well-exposed middle, a hat shape. A dashed overlay shows the SNR-optimal version of the weight, skewed toward the brighter, less-noisy samples rather than symmetric. Below, a sketch of three bracket frames shows each contributing only over the band where its weight is non-zero, with overlap regions (where averaging also denoises) shaded.
fig-hdr-merge-antelope
Figure 17.3.2. The merge on a real bracket (Antelope Canyon). A shorter exposure keeps the bright sunlit slot but buries the rock in noise-dark shadow; a longer exposure lifts the shadows but clips the sun. Each frame's weight map (rainbow, bright means high weight) marks where it is well-exposed, neither clipped nor noisy. The reliability-weighted average in linear radiance keeps the good band from each, and the float radiance map is then tone-mapped for display, so both the sun and the deep shadows read at once.

17.3.3 What comes out is not a picture⧉

The radiance estimate still needs an exposure and rendering choice. A target display may require contrast compression; local tone mapping can compress a coarse base while retaining detail. Edge-aware bases can reduce some halos, but do not guarantee artifact-free output (Chapter 18.3, Two-scale and bilateral tone mapping).

fig-tonemap-gauss-vs-bilateral
Figure 17.3.3. Tone-mapping the Antelope Canyon HDR two ways. Gaussian base: the smoothing crosses the bright-slot edge and lifts a washed-out gray halo along the boundary. Bilateral base: the edge-aware smoothing stops at the boundary, so the rock stays dark and saturated and the edge stays crisp: same range compression, no halo. The zoomed crops isolate the boundary where the difference shows.

17.3.4 Optimizing the capture and merge⧉

A binary validity mask can retain several exposures at once. Among the valid estimates, inverse-variance weights account for unequal precision instead of treating all samples equally.

For two independent unbiased estimates $x,y$ with variances $\sigma_x^2,\sigma_y^2$, we write $\hat L=\beta x+(1-\beta)y$. Minimizing $\beta^2\sigma_x^2+(1-\beta)^2\sigma_y^2$ gives:

$$ \beta^{*} = \frac{\sigma_y^2}{\sigma_x^2 + \sigma_y^2}. $$

Under independence and known positive variances, the minimum-variance linear unbiased estimate weights each observation by inverse variance. Here $\sigma_i^2$ must denote variance after exposure normalization, in radiance units:

$$ \hat L = \frac{\sum_i (I_i/k_i)\,/\,\sigma_i^2}{\sum_i 1/\sigma_i^2}, $$

The preceding result applies to independent estimates with fixed known variances. Saturation and uncertain calibration need separate treatment.

To make those weights concrete, we plug in the camera's noise model. The per-pixel variance is dominated by two terms: photon (shot) noise, whose variance is proportional to the signal and so dominates in the highlights, and read noise, whose variance is constant and so dominates in the shadows:

$$ \sigma^2(x) = a\,x + \sigma_\text{read}^2, $$

where $a$ and $\sigma_\text{read}$ depend on the camera and the ISO. We can calibrate it by shooting repeated frames and measuring the variance across the stack, or derive it on paper. Either way, the merge weights are no longer a hand-drawn hat; they come from the physics of the sensor.

If linear signal $I_i$ has conditional variance $a_i k_iL+b_i$, then the normalized estimate has

$$v_i(L)=\operatorname{Var}(I_i/k_i\mid L)=\frac{a_i k_iL+b_i}{k_i^2},\qquad w_i\propto\frac{1}{v_i(L)}.$$

At common gain with shot noise alone, the relative weights are proportional to $k_i$ for each valid frame. With read noise, the weights generally also depend on scene intensity. We handle that dependence with a pilot radiance estimate or an appropriate likelihood. For correlated measurements, the covariance matrix replaces independent inverse variances. Clipping and data-dependent weights also limit the unbiased-estimator interpretation.

We can measure what each choice of weights does to the noise of the merge on a simulated bracket (Figure 17.3.4). Weights proportional to $k_i$ match the hat almost everywhere and do much better just below the level where a frame clips, where the hat discounts the best sample. In the shadows, where read noise dominates, they trust a short frame too much, and the hat does slightly better there. Only the inverse variance is never worse than the others, and weighting by $1/k_i$, which favors the short frames, is clearly worse than all of them.

fig-hdr-optimal-weights
Figure 17.3.4. Merge weights against the noise of the merged radiance, on a bracket simulated from a radiance map of an office. Each frame is $I_i=\min(1,\,k_iL+n_i)$, with Gaussian noise of variance $a\,k_iL+\sigma_\text{read}^2$ standing in for photon and read noise, drawn once with a fixed seed. Stage 1 plots the SNR of each frame's own estimate $I_i/k_i$ against $\log_2 L$. The next stages merge the frames with binary validity, the Debevec and Malik hat, $w_i\propto k_i$, and the inverse variance $w_i\propto 1/v_i$; all but the last also drop samples below $I_i=0.002$. Lines are the closed form $\operatorname{Var}(\hat L)=\sum_i w_i^2v_i/(\sum_i w_i)^2$, dots the error measured against the noiseless radiance, and the two agree. The weights are evaluated at the noiseless radiance, as a perfect pilot estimate would give them. At the default camera, binary validity loses up to 17 dB where a short frame barely clears the floor, since it trusts that frame as much as a long one. The hat and $w_i\propto k_i$ agree except just below where a frame clips, where the hat discounts the best sample and loses up to 7 dB. In the shadows, read noise makes $w_i\propto k_i$ trust a short frame too much, and there the hat does a little better (15.9 against 15.3 dB measured in the crop); only the inverse variance is best everywhere. Turned on, $w_i\propto 1/k_i$ runs the weighting the wrong way: the crop falls to 6.8 dB, below binary validity's 10.7. Radiance map: Byong Mok Oh. Interactive: step through the stages; click or drag the box on the scene to move the crop; change the number of frames, their spacing, the shortest exposure, the photon gain $a$, the read noise $\sigma_\text{read}$ or the radiance map. Lowering $\sigma_\text{read}$ closes most of the gap between $w_i\propto k_i$ and the inverse variance.

Hasinoff, Durand and Freeman optimize the shutter/ISO sequence for worst-case SNR over a specified brightness range and time budget, using a calibrated camera noise model (Hasinoff et al. 2010). Higher gain can reduce the contribution of downstream read noise on some sensors, at the cost of saturation headroom. The optimum is camera- and scene-dependent; it does not imply a general preference for short, high-ISO exposures. Granados and colleagues develop related noise-aware merging (Granados et al. 2010).

fig-noise-optimal-hdr-5
Figure 17.3.5. The strategy on a real scene: a high-contrast church interior in Nancy. A bracket merged with the noise-optimal capture set recovers both the sunlit windows and the deep shadowed nave in one radiance map, the kind of interior that overflows any single exposure on both ends. What to notice is that the shadows come back clean rather than grainy: the gain in worst-case SNR is exactly what keeps the darkest regions usable Hasinoff et al. 2010.
fig-noise-optimal-hdr-7
Figure 17.3.6. A second real-scene result of the noise-optimal capture strategy. A high-dynamic-range scene shot with the SNR-optimal shutter-and-ISO set and merged, shown against what naive bracketing yields under the same time budget; the optimal capture holds detail across the range with visibly cleaner shadows. The theory transfers to practice: choosing which exposures to shoot, beyond how to weight them, measurably improves the merged result Hasinoff et al. 2010.

The fixed-time capture question also matters for bursts: additional frames incur readout and alignment costs. Chapter 17.6, HDR+ and burst imaging on phones uses a different capture design, retaining short exposures for highlights and motion while combining them for noise reduction.

fig-hasinoff-snr-optimal-set
Figure 17.3.7. The SNR-optimal exposure set (Hasinoff–Durand–Freeman). SNR (in dB) is plotted against log scene brightness for three strategies: standard bracketing at ISO 100, the SNR-optimal high-ISO short-exposure set, and the ideal HDR sensor as an upper bound. The optimal set lifts the worst-case (darkest-region) SNR by roughly 12 dB over naive bracketing and tracks close to the ideal curve across the range, the counter-intuitive result of using higher ISO and shorter exposures once the noise model is taken seriously.
fig-noise-optimal-hdr-6
Figure 17.3.8. The same argument in the authors' own plot, beside our reproduction above. Worst-case SNR is traced against exposure time for each strategy under a fixed time budget: naive ISO-100 bracketing sits well below the SNR-optimal set that favors higher ISO and shorter shutters, which in turn hugs the bound of an ideal HDR sensor. Seeing the published figure makes the claim concrete: once the full photon-plus-read noise model is taken seriously, fewer, shorter, higher-ISO frames win, lifting the darkest-region SNR by roughly 12 dB Hasinoff et al. 2010.

The units of the weight are easiest to see with two exposure times. A long exposure records more photons, but its raw variance is also larger in the shot-noise regime. Dividing each measurement by its exposure time puts both on a radiance scale. The long exposure then has smaller variance on that common scale, because the exposure normalization divides its variance by the square of the time. Weighting raw values by their unnormalized variance would answer the wrong question.

With shot noise alone, the precision of an unclipped radiance estimate grows in proportion to exposure time. Read noise adds a term whose radiance variance decreases with the square of exposure time, making long captures especially useful where that term matters. Clipping removes their usefulness at bright pixels. This is how one frame can be valuable in the shadows and unusable in the highlights without changing the weighting principle.

The capture optimization asks a different question from choosing merge weights. Suppose the total capture time is fixed. Spending most of it on a long exposure gives dark regions more photons, but can leave bright regions clipped in that frame. Dividing it among short exposures provides repeated highlight measurements but incurs read noise repeatedly. Changing gain can alter the contribution of downstream electronics as well as the available headroom. For each candidate sequence, we predict the merged SNR as a function of scene brightness; the objective used by Hasinoff, Durand and Freeman improves the worst part of that curve over a specified range Hasinoff et al. 2010. The figure's comparison should be read as a result for its camera model and budget, rather than as a universal ISO prescription.

The published capture example makes the role of gain concrete. Under a budget of at most three photographs and 0.25 seconds of exposure, the paper's Canon 1D Mark III model compares an ISO-100 bracket of $1/100$, $1/25$ and $1/6$ second with an optimized ISO-3200 sequence of $1/3200$, $1/125$ and $1/5$ second. Its plotted worst-case SNR improves by about 12 dB (project poster). The longest optimized exposure is actually longer than the longest bracket exposure. The result comes from allocating the whole sequence under the sensor's noise and saturation model, not from a rule that increasing ISO and shortening every exposure improves photographs.