13.2 Denoising by averaging⧉
Photograph a gray card several times without changing the camera settings. The value at one pixel fluctuates, but the average becomes more stable as the number of frames grows. Chapter 2.8, Noise, signal-to-noise ratio and dynamic range established the $1/\sqrt N$ law. Here we derive it for a frame stack and examine what happens when motion, outliers, or calibration errors violate its assumptions.
Fix the exposure, gain, and processing; keep the illumination stable; and align corresponding scene points before combining their measurements. Use linear values with the black level removed and preserve negative noise excursions. Under these conditions, independent zero-mean errors average out. They need not be Gaussian or symmetric: an unclipped Poisson count is unbiased even at low light.
Recording several captures lets us choose or combine their measurements afterward. A filter wheel is the plain example: shoot the scene through a red, a green and a blue filter (or through a polarizer at several angles) and the choice of how to combine the bands can be made later, at the desk. Exposure brackets, focal stacks, and panoramas extend this idea to different capture settings.
Several compatible measurements can reduce uncertainty or cover information missing from one frame. This is a different benefit from the previous lesson. Deferring a choice picks among things one exposure could have recorded; combining exposures can exceed what any single exposure could record, such as a noise level below one frame's, a dynamic range wider than the sensor's, or a depth of field deeper than the aperture allows. The reconstruction must account for differences in exposure, focus, viewpoint, or scene motion.
13.2.1 Why averaging works: the $1/\sqrt N$ derivation⧉
Let $X_i=\mu+n_i$ be the measurement of one scene point in frame $i$. The desired signal is $\mu$, and the errors satisfy $\mathbb E[n_i]=0$, $\operatorname{Var}(n_i)=\sigma^2$, and independence across frames. Equal variance is a per-pixel assumption across the stack; it does not require bright and dark pixels to have equal noise.
The estimate is $\bar X=N^{-1}\sum_{i=1}^{N}X_i$:
mean = zero_image()
for frame in aligned_linear_frames:
mean += frame
mean /= number_of_framesIts expected value is
Thus the estimator is unbiased: over repeated stacks, its average equals the true signal. An individual finite stack still contains error. Scaling a random variable by $1/N$ scales its variance by $1/N^2$, and independent variances add, giving
Nothing is blurred by this, which is what separates it from every single-image method in the part: each pixel is averaged only with repeated measurements of itself, never with a different pixel. The cost is in the exponent. Halving the noise again takes four times as many frames, so there are sharp diminishing returns to spending photons this way (Figure 13.2.1).
The signal stays constant while the noise standard deviation falls. Figure 13.2.2 shows the resulting narrowing of a flat patch's histogram.
Averaging four independent measurements of the same signal halves the noise standard deviation. Averaging 100 reduces it by a factor of ten. These statements require zero-mean errors and unchanged per-frame variance. Lowering the effective noise floor can extend the usable range, as described in L2.46, provided the captures have not clipped the signal.
SNR increases by $\sqrt N$, or $10\log_{10}N$ dB: about 3 dB per doubling of the frame count (Figure 13.2.5). This comparison adds capture time while holding each exposure fixed. It does not say that splitting one fixed total exposure into more frames improves SNR.
The same stack estimates the per-frame variance:
The divisor $N-1$ corrects for estimating the mean from the samples whose deviations we measure. For two fair coin flips coded as 0 and 1, the four outcomes give sums of squared deviations $0,\tfrac12,\tfrac12,0$. Dividing by two gives an average estimate of $1/8$, half the true variance $1/4$; dividing by one gives $1/4$.
If errors are correlated, the omitted terms return:
A common offset across frames does not diminish. A fixed calibration bias survives as an error in the mean even when the remaining random fluctuations obey the averaging law.
A four-frame example separates the two assumptions. If the same signal is measured four times with independent errors of standard deviation 8, their average has standard deviation 4. A fixed offset of 3 at that sensor position survives unchanged: averaging adds the offset four times and divides by four. A fluctuation shared by every frame likewise does not receive the independent-noise reduction. Thus a cleaner-looking average can still retain a structured bias. Calibration and averaging solve different parts of the measurement problem, as developed in Chapter 2.8, Noise, signal-to-noise ratio and dynamic range.
The square root also describes the diminishing return on capture time. Halving the noise standard deviation takes four times as many independent, equally exposed frames. Going from one frame to four and from four frames to sixteen buys the same factor of two, but the second improvement costs twelve extra frames. Motion, storage, and latency can make those extra frames more expensive than the noise reduction is worth.
13.2.2 When the plain mean is wrong: robust combination⧉
A satellite crossing one exposure adds a bright streak to the stack. The mean reduces its contribution by $N$, but leaves a visible trace. A per-pixel median is less sensitive to a small number of extreme values (Figure 13.2.6). It can still fail when the wrong scene or moving object supplies most of the samples at that position.
For independent Gaussian samples of a constant signal, the large-sample variance of the median is approximately $\pi/2$ times that of the mean. Its standard deviation is about 1.253 times larger, corresponding to roughly 20% lower SNR. Robust rejection therefore has a cost when all samples follow the assumed Gaussian model.
Sigma-clipping rejects measurements whose deviations exceed a chosen multiple $k$ of an estimated standard deviation. A robust initial center and scale prevent one large outlier from inflating its own rejection threshold. The following is an illustrative procedure for a sufficiently large stack, not a guarantee for a short burst:
for each output pixel p: values = valid_registered_samples(p) if values is empty: mark_output_invalid(p) continue center = median(values) scale = max(1.4826 * median(abs(values - center)), noise_floor(p)) keep = abs(values - center) <= k * scale if too_few_samples(keep): out[p] = center else: out[p] = mean(values[keep])
The factor 1.4826 calibrates the median absolute deviation for Gaussian noise. Choose $k$ and the minimum sample count using the noise model; rejection based on a short stack can discard valid measurements or retain an outlier. Winsorization, which caps extreme values, is distinct from discarding them. Moving boundaries and occlusion usually need alignment confidence and region-dependent weights in addition to a per-pixel test.
For independent unbiased measurements of unequal known variances $v_i$, the minimum-variance linear unbiased estimate uses
Before combining different exposures, convert them to a common signal scale and transform their variances by the square of the same scale factor. Derive weights from calibrated parameters or a stable estimate, since weighting directly by a noisy measured value can bias the result. Noise-optimal HDR capture uses this relationship to select exposures and combine their useful ranges (Hasinoff et al. 2010).
This is the same neighborhood-selection question that appears in edge-aware filtering, now asked across frames. A sample can be nearby after alignment yet belong to a different surface because of motion or occlusion. Rejecting it prevents a ghost, but reduces the amount of independent evidence available there. HDR merging faces the same compromise, with the additional need to reject saturated measurements.
13.2.3 Calibration frames: what averaging can't fix⧉
Averaging does not remove a pattern repeated in sensor coordinates. Bias and dark frames estimate additive offsets; flat fields estimate multiplicative sensitivity variation. The sources and the role of cooling are introduced in Chapter 2.8, Noise, signal-to-noise ratio and dynamic range; the calibration terminology is standard in CCD astronomy (Howell 2006).
Use capped bias frames at the shortest exposure, capped darks at the lights' exposure time, gain, and temperature, and unsaturated flats taken with the same optical configuration. Average enough calibration captures to reduce the uncertainty of each master. A master dark removes expected dark signal, not the independent shot noise that dark current contributed to a light frame. Its own estimation error is reused in every calibrated exposure and can set a floor on the final stack. More light frames do not reduce that shared master error.
Choose an explicit convention. Let $D$ be a matched master dark that includes the bias, and let $F_c$ be the flat after removal of its own appropriate additive calibration. Normalize it as $F_n=F_c/\langle F_c\rangle$, where the mean is taken over valid pixels. Then
Subtract the additive master once, then divide by the relative sensitivity. Do not subtract bias again from $L$ when it is already included in $D$. Flats may require dark-flats rather than only a bias subtraction when their dark signal matters. Reject defective or nearly zero flat values; division there amplifies error (Figure 13.2.7).
The clipping discussion in Chapter 2.8, Noise, signal-to-noise ratio and dynamic range applies directly. At a zero-valued signal, clipping zero-mean Gaussian read noise at zero produces an expected value $\sigma_r/\sqrt{2\pi}$. Averaging reduces the fluctuations around this positive value; it does not amplify or remove the bias. A capture pedestal helps preserve negative read-noise excursions until black-level subtraction. Once those measurements have been clipped, subtracting the pedestal cannot restore them (Figure 13.2.8).
Dithering moves the scene to different sensor positions between captures. After scene registration, a residual sensor defect no longer falls on the same output pixel. This can help rejection and averaging, but does not make every residual independent or zero-mean. Large-scale banding may remain correlated across the dither positions. Calibration should normally precede scene registration because its pattern is defined on the sensor grid.
13.2.4 Handheld low light: the phone in your pocket⧉
A handheld burst replaces one motion-blurred long exposure by several shorter measurements. The merge aligns stable regions and reduces the contribution of measurements that disagree because of motion or occlusion. Near a moving subject, fewer usable samples mean less noise reduction. A single frame count therefore does not describe the quality everywhere in the result.
HDR+ combines burst capture with alignment and noise-aware merging (Hasinoff et al. 2016). Night Sight also estimates motion to choose exposure durations and frame counts, and uses low-light white balance and tone rendering (Liba et al. 2019). These later operations affect the appearance, so a clean-looking output should not be attributed to averaging alone. Chapter 23.3, Focal stacks and depth of field extension considers related capture-time tradeoffs involving aperture and focus (Hasinoff & Kutulakos 2009).
Night Sight makes the capture decision part of denoising Liba et al. 2019. A relatively still scene permits longer individual exposures, reducing how often read noise is incurred for a given light budget. Subject motion favors shorter exposures, which reduce blur within each frame but make each measurement noisier. Motion estimation therefore helps choose exposure durations and frame counts before the burst is combined. Alignment and robust merging then handle disagreement among frames. Low-light white balance and tone rendering address different questions afterward: a low-noise estimate can still have an unwanted cast or look unlike a night scene. The complete result depends on capture, combination, and rendering together.
Recovering a clean signal does not determine how bright a night photograph should look. A sufficiently long exposure can make a moonlit landscape resemble daylight. The Night Sight team drew on illusionistic painting to preserve the impression of night: strong contrast, dark surroundings, and shadows that approach black, even while revealing more color and detail than the eye could see. Their tone curve balances visibility against that impression of darkness. This is a rendering decision after reconstruction, and a useful connection to the discussion of pictorial cues and adaptation in the color and perception parts (Google's account of Night Sight).
13.2.5 Deep-sky astrophotography: averaging at the extreme⧉
A faint nebula may be indistinguishable from noise in one exposure but become visible after hours of integration. Astronomers capture sub-exposures, calibrate them, register the stars, and combine the valid measurements. The exposure of each sub is limited by tracking, saturation, and changing conditions. Satellites, defective pixels, and clouds require rejection or weighting.
For a fixed total integration time $T$ split into $N$ equal exposures, let $s$ be the source electron rate, $b_s$ the sky-background rate, $d$ the dark-current rate, and $r$ the read-noise standard deviation per exposure. Ignoring calibration uncertainty and dead time, the background-subtracted sum has
Total shot-noise variance depends on the total collected electrons; read-noise variance is added once per exposure. Longer subs help until sky, source, or dark-current shot noise dominates read noise. A bright sky generally makes that crossover occur sooner; narrowband imaging with little sky flux may need longer subs. Beyond the crossover, lengthening subs brings little SNR benefit at fixed $T$, while increasing the risks of saturated stars and tracking blur.
The 2019 Pixel astrophotography system used up to fifteen 16-second frames for a four-minute integration, with the phone held stationary on a tripod or another support. It detected hot-pixel outliers and treated the sky separately after segmentation (Google "Astrophotography with Night Sight" 2019). This was a supported multi-frame capture, not a four-minute handheld exposure.
Planetary lucky imaging selects the sharper frames to reduce atmospheric blur, then aligns and stacks the selected frames to reduce noise. Selection and averaging serve different purposes; choosing a smaller sharp subset sacrifices photons for resolution.
A faint nebula makes the repeated-measurement argument visible. A single sub-exposure may contain too little signal for the outer structure to be apparent. Register many subs to the stars and the nebula contributes at consistent scene positions while independent noise fluctuates. Calibration is performed in sensor coordinates before that registration: a fixed defect belongs to the detector, not to a star position. Deliberately shifting the framing between captures can move residual defects relative to the aligned sky and help rejection. The total integration, sky background, read noise, tracking, and saturation together determine which exposure sequence is useful.
13.2.6 When is splitting the light free?⧉
Denoising runs into a floor set by the measurements themselves (Chapter 13.13, The limits of denoising), and the one escape from it is to collect more photons. There is engineering freedom hiding inside that sentence. Given a fixed photon budget (so much light, so much time), does it matter whether you collect it in one exposure or ten? Through one color channel or six? Through the whole aperture or a mosaic of pieces? The affine noise model from Chapter 2.8, Noise, signal-to-noise ratio and dynamic range, $\sigma^2(L) = \sigma_\text{read}^2 + aL$, answers with a clean dichotomy.
Suppose a fixed exposure budget yields $S$ expected electrons in total and each read contributes independent variance $r^2$. One long exposure has $\mathrm{SNR}_1=S/\sqrt{S+r^2}$. Summing $N$ shorter exposures with the same total photon count gives
When $S\gg Nr^2$, splitting has little shot-noise cost. Short exposures can also reduce blur and avoid saturation. Alignment, readout overhead, and changing illumination can still reduce the practical gain Hasinoff et al. 2016.
When read noise dominates, $N$ reads instead of one reduces SNR by up to $\sqrt N$ under these assumptions. This comparison holds total exposure fixed; averaging $N$ frames of unchanged duration instead collects $N$ times as many photons. Exposure and aperture budgets are developed in [@hasinoff-kutulakos-2009; @hasinoff-2010].
The same accounting applies when measurements can be recombined without losing photons or changing the estimated quantity. Absorptive filters and coded masks reduce throughput even in the shot-noise regime. Separating color or depth also creates distinct unknowns, so equal total photon counts alone do not guarantee equal reconstruction quality.
With a fixed total count and negligible read noise, splitting a measurement into independent exposures has little SNR cost if the photons can be recombined. Each additional read adds variance. Lost photons, motion, and new unknowns such as separate depth or color channels add costs beyond this simple accounting.
Chapter 13.2, Denoising by averaging and Chapter 17.6, HDR+ and burst imaging on phones apply this exposure-budget distinction to practical bursts.
Recap: big lessons of this chapter
Recording several captures lets us choose or combine their measurements afterward. A filter wheel is the plain example: shoot the scene through a red, a green and a blue filter (or through a polarizer at several angles) and the choice of how to combine the bands can be made later, at the desk. Exposure brackets, focal stacks, and panoramas extend this idea to different capture settings.
Several compatible measurements can reduce uncertainty or cover information missing from one frame. This is a different benefit from the previous lesson. Deferring a choice picks among things one exposure could have recorded; combining exposures can exceed what any single exposure could record, such as a noise level below one frame's, a dynamic range wider than the sensor's, or a depth of field deeper than the aperture allows. The reconstruction must account for differences in exposure, focus, viewpoint, or scene motion.
Averaging four independent measurements of the same signal halves the noise standard deviation. Averaging 100 reduces it by a factor of ten. These statements require zero-mean errors and unchanged per-frame variance. Lowering the effective noise floor can extend the usable range, as described in L2.46, provided the captures have not clipped the signal.
With a fixed total count and negligible read noise, splitting a measurement into independent exposures has little SNR cost if the photons can be recombined. Each additional read adds variance. Lost photons, motion, and new unknowns such as separate depth or color channels add costs beyond this simple accounting.