Lecture Notes 3 — Noise, bilateral filter🔗⧉

Signal-to-noise ratio: it's about ratios🔗⧉

A counterintuitive fact ties the whole part together. Since shot noise is $\sigma = \sqrt{N}$, the absolute noise is actually larger in the bright parts of the image than in the dark parts: a highlight collecting 10,000 photons has $\sigma = 100$, while a shadow collecting 100 photons has only $\sigma = 10$. And yet noise looks worst in the shadows. The resolution is that noise visibility depends strongly on its size relative to the local signal, as well as on display brightness, spatial frequency, and image content. Signal-to-noise ratio (SNR) captures the first of these effects. For shot noise,

The SNR grows with brightness, so it is worst where the light is faintest (the shadows) even though the raw noise magnitude is smallest there (Figure 2.8.6⛶ fullscreen). This emphasis on ratios also motivates gamma and log encoding (log encoding gives equal steps to equal ratios, while a power-law gamma curve approximates perceptual spacing over a useful range) and exactly why exposing to the right works: collecting more photons everywhere, especially in the shadows, raises $N$ and therefore raises the SNR where it is worst.

One caution before we go on, because the acronyms collide: SNR is not PSNR. The signal-to-noise ratio here is a property of a capture, the true signal at a pixel relative to the sensor's noise there ($\sqrt N$ for shot noise), and it answers how clean is this measurement? Peak signal-to-noise ratio (PSNR), which you meet whenever an algorithm is evaluated, is a different animal: a full-reference quality metric that compares a processed image $\hat I$ against a ground-truth reference $I$, computed from their mean-squared error as

where $\mathrm{peak}$ is the largest possible value (e.g. often 1.0 for us when using floating point values, or 255 for 8-bit) and $\mathrm{MSE}=\tfrac1n\sum(\hat I-I)^2$. So SNR describes a photograph (how much noise a single measurement carries, with no reference needed), while PSNR scores a reconstruction against a known truth (how close a denoiser or codec got). Both compare a signal scale to an error scale, but their numerators and purposes differ. PSNR uses a fixed peak rather than the local signal. Maximizing expected PSNR through minimizing MSE favors a conditional-mean estimate, which can blur uncertain detail; that makes it useful for some comparisons but incomplete as a perceptual score.

A word on the decibel itself, since it recurs through this chapter and the next. A decibel is just a logarithmic way to write a ratio: for a power (or energy, intensity, variance) ratio it is $10\log_{10}$, and for an amplitude (or voltage, signal) ratio it is $20\log_{10}$, the factor of two coming from power $\propto$ amplitude$^2$. Photographers count the same ratios in stops, factors of two ($\log_2$), so one stop is about 6 dB of amplitude or 3 dB of power. Decibels, stops, plain linear ratios, and scientific notation are thus four scales for the same underlying number, a multiplicative factor, and it helps to move fluently between them (Figure 2.8.5⛶ fullscreen).

The affine model, whose full story is the next section, settles a lot of everyday "will this help my noise?" questions at a glance. Read each move against $\sigma^2 = g\,I + \sigma_r^2$ (signal $I$, photon-gain slope $g$, read-noise floor $\sigma_r$), tracking what happens to the signal, the noise $\sigma$, and their ratio. The entries below are for the shot-noise-dominated regime (well-lit midtones); the read-noise term $\sigma_r$ changes only the two rows that mention it.

Software rescaling preserves SNR. Analog gain can reduce the relative contribution of noise added downstream, as the ISO section explains. Combining $N$ equally exposed independent measurements improves SNR as $\sqrt N$, while more photons per measurement improve the capture itself.

PSNR🔗⧉

Peak signal-to-noise ratio expresses the same error in decibels:

Here $L$ is the fixed reference range: $1$ for normalized samples in $[0,1]$, or $255$ for 8-bit samples. It is not each image's measured maximum. For HDR values without a shared finite range, specify a normalization or use a measure designed for the intended luminance conditions. PSNR is not the sensor SNR discussed in Chapter 2.8, Noise, signal-to-noise ratio and dynamic range: the denominator here includes any discrepancy from the reference, including bias, blur, and registration errors.

Higher PSNR means lower MSE at the same $L$; zero error gives infinity. Halving MSE adds about 3 dB. There is no universal threshold for an acceptable image. Scores depend on the scene, encoding, crop, and degradation.

Be explicit about aggregation. Averaging image PSNRs differs from pooling squared errors and converting once, because the logarithm is nonlinear. For equal-sized images with MSEs $0.001$ and $0.01$ at $L=1$, the mean PSNR is 25 dB, while the PSNR of their mean MSE is about 22.60 dB. Report the convention and inspect the distribution, not only its mean.

Robustness🔗⧉

With the Gaussians defined above and the center included, $w(p,p)=1$, so $W_p\geq1$. The normalized bilateral is a convex average and cannot blow up from division by a near-zero normalizer. An isolated value can still be poorly denoised because it assigns little weight to other samples. This is a limitation of the affinity, not proof that normalization is numerically invalid.

The deeper weakness is that the range term reads the noisy image itself. At an outlier the center's own corrupted value decides which neighbors count as similar, so the pixel that most needs help is the one whose weights are least trustworthy: it finds nothing that resembles it and averages with almost nothing but itself. A hot pixel survives the filter for the same reason an edge does.

A cheap repair is to compute the range term from a pre-filtered copy instead of from the input. Run a small median filter first, then use that image as the reference for the value differences while still averaging the original samples. The median removes impulses without moving an edge (Chapter 13.3, Gaussian, median, and bilateral filtering), so the weights are decided by something close to the clean signal while the output remains an average of real measurements. Mechanically this is the cross-bilateral construction of the next section, with the guide derived from the input rather than captured separately.

The cost is that the guide's failures become the filter's. A median erases thin lines, small dots and isolated highlights, and a structure the guide has removed has no range weight left to defend it: the bilateral will happily average it away. Check the guide on the structures you care about before trusting the result, and keep the median's window small enough that it removes impulses rather than detail.

An alternative discussed by Aubry et al. is an unnormalized correction Aubry et al. 2014. Its scale must be defined. With these unnormalized spatial weights, introduce a step size $\eta$:

Each neighbor contributes a weighted difference from the center. A sufficient condition for this update to remain a convex combination is $0\leq\eta\sum_{q\ne p}w(p,q)\leq1$. One practical convention normalizes the spatial kernel to sum to one and uses $0\leq\eta\leq1$. Without a scale convention, adding more neighbors can cause overshoot. The update generally differs from the normalized bilateral; neither is a universal replacement for the other.

This unnormalized form is also the bilateral cousin of the local Laplacian filters: working in deltas from the input, scale by scale, is exactly the move that makes the multiscale local Laplacian fast and robust (Paris / Aubry; cross-ref Chapter 10.4, Local Laplacian filters and Chapter 6.6, Pyramids and wavelets).

Cross / joint bilateral🔗⧉

So far the affinity has been read off the same image we are smoothing (the range weight uses $I$ itself). But there is no reason it must. The cross (or joint) bilateral filter reads the range weight from a different image, a guide $J$, while still averaging the target $I$:

The averaging runs over the target $I$; the affinity comes from the guide $J$. Why want that? Because sometimes the image you wish to smooth is too noisy or too dark for its own edges to be trusted. If you read the affinity off $I$, you are reading off the noise. If you hold a cleaner signal that shares the same scene geometry, borrow its edges instead.

The headline application is flash / no-flash photography (Petschnigg et al. 2004; Eisemann & Durand 2004). Shoot a dim scene twice: once without flash (the right mood and color, but dark and noisy) and once with flash (clean and well-exposed, but flat, harsh lighting). After registration, many object boundaries correspond. Motion, occlusion, flash shadows and specularities can introduce mismatched edges, so a practical pipeline needs confidence masks or fallback behavior. So smooth the noisy no-flash image using the flash image as the guide: the flash image's clean edges drive the range weight, telling the filter where it may and may not average, while the values being averaged come from the ambient no-flash shot. The result keeps the no-flash mood and kills the no-flash noise (Figure 10.1.8⛶ fullscreen). We develop the full flash/no-flash pipeline later (forward-ref → Flash / no flash, computational illumination).

The full flash/no-flash story, including Petschnigg's detail transfer, Eisemann & Durand's intrinsic-relighting pipeline, flash-shadow removal, and their end-to-end results, belongs with the rest of computational illumination, where capturing the scene under two lights is the whole subject (Flash / no flash).

The cross bilateral is the first instance of a more general joint / guided idea (filter one image using structure taken from another) that culminates in the guided filter (later in the part).

Coarse-to-fine alignment on a pyramid🔗⧉

Brute force has two failure modes that bite together. Large motion forces a large, and therefore expensive, search window; and a raw SSD surface is not always the clean single-welled basin of the idealized figure. Repetitive texture (a brick wall, a picket fence) and aliasing carve local minima into it, and a local refinement started far from the answer can fall into the wrong well. An exhaustive search finds the best score in its window, though that score may favor a false match. Going multiscale addresses both, though only the first reliably (Figure 12.1.6⛶ fullscreen).

Build a Gaussian pyramid of both frames and align coarse-to-fine (the pyramid of Chapter 6.6, Pyramids and wavelets, here repurposed as a search strategy). At the tiny, heavily-blurred top of the pyramid, a small search in pixels covers a large real motion, because each coarse pixel stands for many fine ones, so large displacements become cheap. Better still, the low-frequency surface at the top may be smoother because blurring attenuates high-frequency detail, often making a rough estimate easier to obtain. Propagate that estimate down to the next level, scale it up, and refine it with a tiny local search; the high frequencies you add as you descend mostly refine the estimate, because the local search at each level is too small to jump to a distant minimum. That is also the risk: if the coarse estimate is wrong, the finer levels rarely recover.

Pyramids can smooth the cost and reduce a large displacement to a smaller coarse-grid shift. They do not guarantee one clean basin: periodic texture and lost small structures can remain ambiguous. Verify the refined result on the original images.

Build each level by filtering before decimation. Otherwise the coarse image can contain alias patterns that disagree between frames, creating false evidence for motion. A pyramid is useful both as a representation and as a search strategy: solve large motions on a coarse grid, then recover precision from finer samples. HDR+ applies this idea to tiles before robust burst merging Hasinoff et al. 2016. Local refinement should still be checked against alternatives when the coarse choice is ambiguous.

When the plain mean is wrong: robust combination🔗⧉

A satellite crossing one exposure adds a bright streak to the stack. The mean reduces its contribution by $N$, but leaves a visible trace. A per-pixel median is less sensitive to a small number of extreme values (Figure 13.2.6⛶ fullscreen). It can still fail when the wrong scene or moving object supplies most of the samples at that position.

For independent Gaussian samples of a constant signal, the large-sample variance of the median is approximately $\pi/2$ times that of the mean. Its standard deviation is about 1.253 times larger, corresponding to roughly 20% lower SNR. Robust rejection therefore has a cost when all samples follow the assumed Gaussian model.

Sigma-clipping rejects measurements whose deviations exceed a chosen multiple $k$ of an estimated standard deviation. A robust initial center and scale prevent one large outlier from inflating its own rejection threshold. The following is an illustrative procedure for a sufficiently large stack, not a guarantee for a short burst:

for each output pixel p:
    values = valid_registered_samples(p)
    if values is empty:
        mark_output_invalid(p)
        continue
    center = median(values)
    scale = max(1.4826 * median(abs(values - center)), noise_floor(p))
    keep = abs(values - center) <= k * scale
    if too_few_samples(keep):
        out[p] = center
    else:
        out[p] = mean(values[keep])

The factor 1.4826 calibrates the median absolute deviation for Gaussian noise. Choose $k$ and the minimum sample count using the noise model; rejection based on a short stack can discard valid measurements or retain an outlier. Winsorization, which caps extreme values, is distinct from discarding them. Moving boundaries and occlusion usually need alignment confidence and region-dependent weights in addition to a per-pixel test.

For independent unbiased measurements of unequal known variances $v_i$, the minimum-variance linear unbiased estimate uses

Before combining different exposures, convert them to a common signal scale and transform their variances by the square of the same scale factor. Derive weights from calibrated parameters or a stable estimate, since weighting directly by a noisy measured value can bias the result. Noise-optimal HDR capture uses this relationship to select exposures and combine their useful ranges (Hasinoff et al. 2010).

This is the same neighborhood-selection question that appears in edge-aware filtering, now asked across frames. A sample can be nearby after alignment yet belong to a different surface because of motion or occlusion. Rejecting it prevents a ghost, but reduces the amount of independent evidence available there. HDR merging faces the same compromise, with the additional need to reject saturated measurements.

Handheld low light: the phone in your pocket🔗⧉

A handheld burst replaces one motion-blurred long exposure by several shorter measurements. The merge aligns stable regions and reduces the contribution of measurements that disagree because of motion or occlusion. Near a moving subject, fewer usable samples mean less noise reduction. A single frame count therefore does not describe the quality everywhere in the result.

HDR+ combines burst capture with alignment and noise-aware merging (Hasinoff et al. 2016). Night Sight also estimates motion to choose exposure durations and frame counts, and uses low-light white balance and tone rendering (Liba et al. 2019). These later operations affect the appearance, so a clean-looking output should not be attributed to averaging alone. Chapter 23.3, Focal stacks and depth of field extension considers related capture-time tradeoffs involving aperture and focus (Hasinoff & Kutulakos 2009).

Night Sight makes the capture decision part of denoising Liba et al. 2019. A relatively still scene permits longer individual exposures, reducing how often read noise is incurred for a given light budget. Subject motion favors shorter exposures, which reduce blur within each frame but make each measurement noisier. Motion estimation therefore helps choose exposure durations and frame counts before the burst is combined. Alignment and robust merging then handle disagreement among frames. Low-light white balance and tone rendering address different questions afterward: a low-noise estimate can still have an unwanted cast or look unlike a night scene. The complete result depends on capture, combination, and rendering together.

Recovering a clean signal does not determine how bright a night photograph should look. A sufficiently long exposure can make a moonlit landscape resemble daylight. The Night Sight team drew on illusionistic painting to preserve the impression of night: strong contrast, dark surroundings, and shadows that approach black, even while revealing more color and detail than the eye could see. Their tone curve balances visibility against that impression of darkness. This is a rendering decision after reconstruction, and a useful connection to the discussion of pictorial cues and adaptation in the color and perception parts (Google's account of Night Sight).

The median, and the noise it is actually for🔗⧉

The median selects the middle value of a sorted neighborhood. It is also a minimizer of

Changing an outlier's magnitude can move the mean arbitrarily far, but need not move the median. For example, the values $10,10,11,11,255$ have mean 59.4 and median 11. This makes the median useful for sparse impulse corruption (Figure 13.3.3⛶ fullscreen).

For a constant signal with independent Gaussian errors, the sample median's large-sample standard deviation is about $\sqrt{\pi/2}=1.253$ times the mean's. Robustness is useful when outliers are present, but has an efficiency cost when they are absent.

A median can preserve an ideal step when the correct side supplies a majority of the window. The same majority rule removes thin lines, isolated stars, and small details, and can round corners. It is nonlinear, so it has no single convolution kernel or frequency response. Repeated passes are not equivalent to one pass with a larger window. Iterating one median does, however, stop changing things: for a finite 1-D signal, repeated passes converge to a root signal that the filter leaves unchanged, made of constant runs and monotone stretches long enough to survive the window Gallagher & Wise 1981. Further passes then do nothing.

The median's majority rule explains both a strength and a failure. In a window containing mostly dark background and one extreme bright outlier, the middle sorted value remains dark. That suppresses a hot pixel, but it can also suppress a real isolated star or a thin bright line. At a broad ideal step, the side occupying most of the window determines the answer. The filter has no knowledge of which small structure the photographer intended to preserve. Inspect small real details as well as the removed outliers when choosing its window.

Denoise color more than brightness🔗⧉

Vision generally resolves luminance detail more finely than chromatic detail. Many denoisers therefore smooth chroma more strongly. This is a tolerance, not permission to erase arbitrary color structure: isoluminant colored edges and fine saturated patterns can be visible.

A luminance/chrominance representation provides separate controls for brightness grain and color blotches. Stronger chroma smoothing can clean a neutral wall while retaining luminance texture; across colored boundaries it can bleed colors. RGB methods can also learn or implement such coupling. Choose the representation and weights for the image, rather than treating one channel layout as necessary for good denoising.

What each filter destroys🔗⧉

Compare results on flat regions, edges, and texture. Matching residual noise in a constant patch gives one useful operating point, but does not determine a universal ranking. An impulse-removal test and a Gaussian-noise test reward different behavior. Also inspect the removed signal $y-\hat x$: recognizable structures suggest lost detail, while their absence does not prove the estimate is correct.

Gaussian smoothing blurs boundaries. The median suppresses minority features along with outliers. The bilateral filter depends on contrasts being distinguishable from noise. These are consequences of the estimators, not parameters that can always be tuned away.

With every filter of this chapter on the table, the most useful exercise is to run them against the same noisy image and watch the trade-offs move. The lab below splits one image into four panes, the split following the mouse, each pane with its own filter, its own sliders and its own live score (Figure 13.3.7⛶ fullscreen). Put the Gaussian beside the bilateral and slide across an edge; set a pane to the amplified error against the clean original to see where each filter cheats, or to the difference of two panes to compare them directly.

The limitation of a small window motivates the next methods. A faint texture may have too little evidence locally to distinguish it from noise. Multiscale methods look for related structure across scales; non-local methods search for repeated patches farther away; learned denoisers use regularities acquired from other images. Each broadens the evidence available to the estimate, while introducing its own assumptions about which structures should agree.

Non-local means: similarity as the weight🔗⧉

Non-local means (Buades et al. 2005) averages observed values with weights derived from patch similarity. One common version compensates for the expected noise contribution:

Here $S_p$ is a bounded search window and $h>0$ has the same intensity units as $\sigma$. The normalization by $m$ in the distance is essential: subtracting $2\sigma^2$ from an unnormalized sum would use the wrong noise scale. The subtraction and truncation are a practical weighting rule, not an unbiased estimator of similarity.

for p in image:
    w = empty_map()
    candidates = search_window(p) excluding p
    for q in candidates:
        d2 = mean((patch(y,p) - patch(y,q))**2)
        w[q] = exp(-max(d2 - 2*sigma**2, 0) / h**2)
    if candidates is empty:
        out[p] = y[p]
        continue
    w[p] = max(w[q] for q in candidates)
    if sum(w.values()) == 0:
        out[p] = y[p]
        continue
    out[p] = sum(w[q]*y[q] for q in candidates plus p) / sum(w.values())

This self-weight convention prevents an exact self-match from overwhelming much smaller neighboring weights. If the candidate set is empty or all weights underflow to zero, return $y_p$. Use a consistent boundary extension when extracting patches. For color images, distances can combine channels and the same weights can filter all channels, with the noise model adjusted for their variances and correlations.

Larger search windows cost more and offer more accidental matches as well as useful ones. Larger $h$ accepts less similar patches and smooths more strongly. Patchwise variants estimate whole patches and aggregate their overlaps. These details distinguish implementations of Chapter 10.2, Non-local means that otherwise share the same equation.

The method also carries a bias. It averages a patch with the patches that resemble it, so it pulls the image slightly toward its own repeated structures, and details with no match elsewhere, a crack in one brick, a single hair, get fewer effective samples and either stay noisy or are smoothed toward their nearest look-alikes. This is the first appearance of a failure mode that every method borrowing evidence from elsewhere in the image shares: good at what repeats, weaker at what does not.

The direct search repeats much of its arithmetic. For a fixed displacement, form an image of squared differences between the input and its shifted copy. A box sum, implemented with a summed-area table Crow 1984 or separable running sums, then gives all patch distances for that displacement. This reduces the cost of comparing large patches. A bounded search window limits the number of displacements; a pilot denoising pass can help select candidates, provided the final average still uses the intended measurements. These accelerations change different parts of the computation and can be combined.

BM3D: group, transform, shrink, aggregate🔗⧉

BM3D (Dabov et al. 2007) combines patch matching with Chapter 13.4, Pyramid and wavelet denoising. For each reference patch it selects similar patches, stacks them, applies a transform within each patch and along the stack, shrinks the coefficients, and transforms back. The third dimension indexes patches; it is not a spatial axis in the photograph.

If the patches were identical, an orthonormal transform along the stack with a constant first basis vector would concentrate their shared signal in that component. Real matches differ, so other components also contain signal. Shrinkage can retain these differences when they are sufficiently strong. It can also erase a genuine detail present in only one patch.

The method has two passes. The first uses hard thresholding and aggregates the reconstructed patches into a basic estimate. The second uses that estimate to select groups and estimate coefficient power, but filters groups extracted from the original noisy image. In a simplified orthonormal-transform description, its empirical Wiener gains are

A small pilot coefficient gives a small gain; a large one largely retains the noisy coefficient. With other transform normalizations, replace $\sigma^2$ by the appropriate coefficient noise variance. Group selection and overlapping samples make the actual statistics more complicated than this idealized expression.

basic = aggregate(
    inverse_transform(hard_threshold(transform(group_from(y)), threshold))
    for each reference patch, with group matches found in y)

for each reference patch:
    positions = find_matches(basic)
    c_basic = transform(extract_group(basic, positions))
    c_noisy = transform(extract_group(y, positions))
    gains = abs(c_basic)**2 / (abs(c_basic)**2 + coefficient_noise_variance)
    accumulate(inverse_transform(gains * c_noisy), positions, group_weights)
out = accumulated_values / accumulated_weights

This outlines the data flow; a reproducible implementation must also specify transforms, matching thresholds, group sizes, spatial windows, and aggregation weights. Every reconstructed patch returns to its own original position. Overlapping estimates are combined with spatial and group reliability weights, then normalized. They share input pixels and are not independent new observations.

Suppose four matched patches were exactly equal except for independent noise. Along the stack dimension, their common component is constant, while their differences contain only noise. An orthonormal transform separates that constant direction from difference directions. Shrinking the latter suppresses disagreement while retaining the common patch. Real matches are only approximate: a small edge displacement or a texture difference can also occupy a difference direction. BM3D's grouping, shrinkage, and overlap aggregation must therefore balance agreement against genuine variation. This explains why grouping quality matters as much as the transform itself.

Learned denoisers🔗⧉

Wavelet shrinkage assumes that useful image structure has a sparse representation. BM3D Dabov et al. 2007 searches for repetition within the observed photograph. A learned denoiser fits a mapping from noisy images to clean targets, using regularities across a training collection as well as spatial context within each input. Its behavior depends on the training images, corruption model, architecture, and loss.

For paired training with squared error, the ideal population solution is the posterior mean:

This characterizes an unrestricted optimum. Finite networks, finite data, and imperfect optimization only approximate it. A network can also be trained with another loss, in which case the optimal statistic changes. These distinctions matter when a trained denoiser is later interpreted as a score estimator in Chapter 13.11, Denoising as a universal prior.

Generative denoising with diffusion models🔗⧉

In a very noisy photograph of a brick wall, several positions of a mortar line may be compatible with the measurement. Averaging those possibilities produces a softened edge. Choosing one produces a sharper edge whose position may be wrong. Generative denoising makes this choice explicit by producing plausible reconstructions conditioned on the noisy image.

The distinction matters where the observation leaves substantial uncertainty. It is not true that all posterior means look blurred or all posterior samples look realistic. Both depend on the scene, measurement, and learned distribution. A model that excludes the true structure can produce confident, consistent errors.

Denoising TLDR🔗⧉

For ordinary photographic work, start with a pretrained denoiser designed for your input. The practical choices are what to capture, how much detail to retain, and how to judge the result.

• Preserve raw data when available. Match the denoiser to the camera data and processing domain it expects; a model for raw sensor noise and one for processed RGB solve different problems. In a custom pipeline, check black levels, image scaling, and noise units before tuning strength (Chapter 13.1, Noise models, transforms, and benchmarks).

• Use multiple frames when motion and capture time permit. Four aligned, independent exposures of unchanged duration halve noise standard deviation. Moving subjects and occlusions reduce that gain locally; splitting a fixed total exposure adds read noise. Use a temporal method for video and inspect ghosting, flicker, and delay (Chapter 13.2, Denoising by averaging, Chapter 13.10, Video denoising).

• Set strength by the detail you need to preserve. Inspect hair, foliage, fabric, small text, and isolated highlights at full resolution and at the intended display size. Some remaining grain is preferable to erased texture. More plausible texture from a generative method does not establish that the camera recorded it. Keep the original and record whether the output uses generative reconstruction (Chapter 13.9, Generative denoising with diffusion models).

• Test on your own images. Compare identical inputs, inspect the removed residual, and use reference error and perceptual measures when a reliable reference exists. Include difficult scenes and measure latency and memory. A benchmark winner may behave differently on your camera (Chapter 13.12, Measuring a denoiser).

• For implementation, use BM3D Dabov et al. 2007 as a baseline requiring no learned weights. Reserve median filtering for sparse impulses or defects; ordinary spatial smoothing also removes detail. Train a new model when available models fail your domain. Noisy-target training can help, but requires the independence and alignment assumptions developed in Chapter 13.8, Denoising without clean data.

Recap: big lessons in this lecture🔗

💡 Big lesson (L2.41) — bright pixels are noisier (more absolute noise) but have better SNR🔗

Shot noise is Poisson, so its variance grows linearly with the signal ($\sigma^2 \propto N$) and its standard deviation grows only as the square root ($\sigma \propto \sqrt{N}$). Two consequences pull in opposite directions: the absolute noise is larger in highlights than in shadows (a 10,000-photon highlight has $\sigma=100$; a 100-photon shadow has $\sigma=10$), yet the SNR $= N/\sqrt{N} = \sqrt{N}$ is higher in the highlights: the signal outgrows the noise. So bright areas are noisier in absolute terms but cleaner to the eye; the shadows have the worst SNR (Figure 2.8.4, Figure 2.8.6⛶ fullscreen). This is why exposing to the right and brighter scenes look smoother, and why we gamma/log-encode so equal ratios get equal steps.

💡 Big lesson (L10.1) — edge-preserving = affinity🔗

Use the color / intensity difference between two pixels as how much they belong together, an affinity, then average (or connect) pixels in proportion to it. Pixels with high affinity are treated as measurements of the same underlying thing and get averaged together; pixels with low affinity, across an edge, are kept apart. The bilateral filter's range weight $g_r$ is the first full instance, but the same affinity drives the bilateral grid, the guided filter, non-local means (NL-means), colorization, and matting / segmentation that follow. Edge-preserving is affinity. (This is L10.1show, registered in Part 13, Denoising where the bilateral first appeared for noise; this is its full edge-preserving treatment.)

💡 Big lesson — define the scale of an unnormalized update

Removing the bilateral normalizer changes the filter. Specify a normalized spatial kernel or an explicit step size, then test constants, impulses and edges. A small total affinity makes a small correction; it does not establish that an isolated value is correct.

💡 Big lesson (L13.1) — capture the full set, decide later🔗

Recording several captures lets us choose or combine their measurements afterward. A filter wheel is the plain example: shoot the scene through a red, a green and a blue filter (or through a polarizer at several angles) and the choice of how to combine the bands can be made later, at the desk. Exposure brackets, focal stacks, and panoramas extend this idea to different capture settings.

💡 Big lesson (L13.2) — multiple-exposure imaging breaks the limits of a single capture🔗

Several compatible measurements can reduce uncertainty or cover information missing from one frame. This is a different benefit from the previous lesson. Deferring a choice picks among things one exposure could have recorded; combining exposures can exceed what any single exposure could record, such as a noise level below one frame's, a dynamic range wider than the sensor's, or a depth of field deeper than the aperture allows. The reconstruction must account for differences in exposure, focus, viewpoint, or scene motion.

💡 Big lesson (the $1/\sqrt N$ law)

Averaging four independent measurements of the same signal halves the noise standard deviation. Averaging 100 reduces it by a factor of ten. These statements require zero-mean errors and unchanged per-frame variance. Lowering the effective noise floor can extend the usable range, as described in L2.46show, provided the captures have not clipped the signal.

💡 Big lesson (local denoising trades noise reduction against signal mixing)

A larger neighborhood supplies more samples but may mix different scene values. Edge-aware and robust rules reduce some mixing errors; they do not establish that every retained feature is signal or every removed feature is noise.