13.3 Gaussian, median, and bilateral filtering🔗⧉

A noisy photograph of a wall contains nearby pixels with similar clean values. Averaging them reduces noise. At the wall's boundary, however, the same operation mixes two surfaces and blurs the edge. Single-image local denoising must decide which neighbors can supply useful evidence about a pixel.

Everything in this chapter is a different answer to one question: which neighbors count as measurements of the same value? In a flat region every neighbor qualifies and averaging cleans up beautifully. At the boundary between a dark cake and a bright wall the neighbors on the far side are measurements of something else, and averaging them smears the two into a muddy band.

We compare a fixed Gaussian average, a median, and a bilateral average whose weights depend on the measured values. Their assumptions differ, and none guarantees preservation of every small feature. The derivation below uses independent zero-mean noise of constant variance in one channel; raw signal-dependent noise requires calibrated local variances or stabilization (Chapter 13.1, Noise models, transforms, and benchmarks).

13.3.1 Gaussian smoothing, and the assumption inside it🔗⧉

Averaging a neighborhood assumes that every pixel in it is a measurement of the same underlying value, corrupted independently (the image is locally constant). Where that holds, the average is the right estimate and the noise cancels out. Where it fails, in particular at any boundary between two surfaces, the average mixes two different values and returns something that was never in the scene. A Gaussian kernel softens the assumption by trusting near neighbors more than far ones, but it still applies the same fixed weights at every pixel, whatever the image is doing there.

For analysis, consider a fixed clean image $x$. What varies is the measurement: the measured value is the truth plus an error, $y_q = x_q + n_q$, and the error is a fresh draw every time the shutter opens (Chapter 13.1, Noise models, transforms, and benchmarks).

For now, we assume two important properties of those draws. First, they average out, meaning the error at a pixel tends to zero as the number of shots increases. Second, they are independent between pixels: knowing that a photosite read high says nothing about its neighbor. That is true of shot noise and read noise, and false of fixed-pattern response and of noise that has already been through demosaicking or resampling. The variance $\sigma^2$ is the size of a typical error, squared; a refresher on these terms is in Refreshers#Probability and information theory.

Because noise is probabilistic, we analyze denoising through the thought experiment of shooting the same scene a large number of times with different noise instances each time. We then conceptually run the same filter on each frame and look at one output pixel across all the different results. We then characterize the performance of the method via statistical estimates such as the average squared error. This is related to the denoising by averaging technique studied in Chapter 13.2, Denoising by averaging but here the goal is to understand the expected error in one run via a probabilistic average error analysis, not to actually compute averages to reach the value of a pixel.

The average error can manifest in two main ways. The answers can be scattered, which means the filter still passes noise through. And their center can sit away from the truth, which means the filter has mixed in something that was never at this pixel. The first is the variance of the estimate, the second is its bias, and the error that matters, squared and averaged, is the bias squared plus the variance. A denoiser is an attempt to make that sum small.

A normalized Gaussian kernel supplies fixed nonnegative weights $w_{pq}$ that decrease with distance from the output pixel $p$:

For independent errors of equal variance $\sigma^2$, the output variance and bias are

The bias formula is important because it is the only place the noise-free image itself appears. The term $x_q-x_p$ captures the deviation from the assumption of a locally constant image. Let's look at variance and bias on two patches. On a flat patch every $x_q$ equals $x_p$, so every term of the bias is zero, and with a box of nine equal weights the variance is $\sigma^2/9$: the noise standard deviation falls by three, and nothing is lost. Now slide that same window onto an edge where the far side is 100 units brighter, with half the weight landing across it. The variance is unchanged, still $\sigma^2/9$, while the bias is about 50 units, next to a noise level of perhaps two or three. The filter has not failed at its job; it has done exactly what it was told, on an assumption that stopped being true.

This is called the bias-variance tradeoff. Widening the kernel moves both terms at once, buying variance with bias, and no choice of width escapes the trade: at any setting, a fixed kernel that is wide enough to clean the flat regions is wide enough to damage the edges. That is why the chapters after this one change the weights rather than the width, why Chapter 13.12, Measuring a denoiser compares two denoisers at a matched operating point rather than at matched parameters, and what Chapter 13.13, The limits of denoising pushes to its bound.

Define the effective sample count $N_{\mathrm{eff}}=1/\sum_qw_{pq}^2$, which is the number of independent samples the weighted average is worth. Equal weights over $N$ independent samples give $N_{\mathrm{eff}}=N$ and recover the averaging law of Chapter 13.2, Denoising by averaging; a Gaussian gives rather fewer than its tap count, since the outer taps contribute little.

Choosing the width is therefore choosing what to sacrifice. Set $\sigma_s$ from the size of the structure you are willing to lose: everything substantially smaller than the kernel is on its way out, and everything substantially larger survives with its edges softened. In practice the Gaussian is also the cheap choice, because it is separable and a two-dimensional blur costs two one-dimensional passes (Chapter 6.2, Convolution).

Start with the Gaussian alone in that panel, and set the other two aside until the sections that introduce them. Widen it and watch the two terms trade: the flat patch cleans up, the edge softens, and the residual view fills with the structure the filter took.

The frequency picture says the same thing and explains why blurring is worth doing at all. White noise has a flat power spectrum, so it carries as much energy at high frequencies as at low. Photographs do not: most of their energy sits at low frequencies, and the fine detail that occupies the high end is a small part of the total. A low-pass filter therefore discards a large share of the noise and a small share of the signal. That argument has an optimum, and the optimum is not a Gaussian.

The Wiener filter Wiener 1949 is what the frequency argument looks like when it is pushed to its conclusion: keep each frequency in proportion to how much of it is signal rather than noise. Its assumptions are the catch. It needs both spectra in advance, and it treats the image as one stationary process, when a photograph is a collection of regions with different statistics. A filter that could vary its behavior by region would do better, which is the move the rest of the chapter makes.

13.3.2 The median, and the noise it is actually for🔗⧉

The median selects the middle value of a sorted neighborhood. It is also a minimizer of

Changing an outlier's magnitude can move the mean arbitrarily far, but need not move the median. For example, the values $10,10,11,11,255$ have mean 59.4 and median 11. This makes the median useful for sparse impulse corruption (Figure 13.3.3⛶ fullscreen).

For a constant signal with independent Gaussian errors, the sample median's large-sample standard deviation is about $\sqrt{\pi/2}=1.253$ times the mean's. Robustness is useful when outliers are present, but has an efficiency cost when they are absent.

A median can preserve an ideal step when the correct side supplies a majority of the window. The same majority rule removes thin lines, isolated stars, and small details, and can round corners. It is nonlinear, so it has no single convolution kernel or frequency response. Repeated passes are not equivalent to one pass with a larger window. Iterating one median does, however, stop changing things: for a finite 1-D signal, repeated passes converge to a root signal that the filter leaves unchanged, made of constant runs and monotone stretches long enough to survive the window Gallagher & Wise 1981. Further passes then do nothing.

The median's majority rule explains both a strength and a failure. In a window containing mostly dark background and one extreme bright outlier, the middle sorted value remains dark. That suppresses a hot pixel, but it can also suppress a real isolated star or a thin bright line. At a broad ideal step, the side occupying most of the window determines the answer. The filter has no knowledge of which small structure the photographer intended to preserve. Inspect small real details as well as the removed outliers when choosing its window.

13.3.3 The bilateral filter: a neighborhood that looks at the pixels🔗⧉

Put the two previous sections side by side and the bilateral filter is what falls out of their complaints.

The Gaussian failed by bias at a discontinuity. Its weights are decided by geometry alone, so at an edge it pours the far side into the answer, and widening it to clean the flat regions only makes that worse. The median answered the same edge by refusing to average at all: order the neighborhood, take the middle value, and a minority of unlike samples cannot move it. That buys robustness and pays for it twice. One order statistic throws away the noise reduction that averaging gives, so on a flat patch the median is the weaker estimator, and the majority rule that protects the edge deletes whatever the majority disagrees with.

The bilateral filter keeps the averaging and changes what decides the weights. Like the Gaussian it sums many neighbors, so noise still falls with the number of samples that genuinely contribute, which is the accuracy the median gave up. Like the median it declines to be dragged by values that do not belong. What differs is the reference. The median compares a neighbor against the rest of the neighborhood and keeps whatever wins the vote; the bilateral compares each neighbor against the center pixel's own measurement, so a neighbor earns its weight by agreeing with this pixel rather than with the crowd. The measurement at the pixel being filtered counts for more than all the others put together, which is what none of the fixed rules allows.

The difference is invisible on a straight edge, where half the window agrees and any sensible rule survives, and unmistakable at a corner. A pixel at the corner of a bright roof against dark sky sees roughly a quarter of its neighborhood belonging to its own surface and three quarters belonging to the other. Every rule that trusts the majority answers with the sky: the median rounds the corner off, which is exactly the failure the previous section described, and the Gaussian smears it. The bilateral asks whether a neighbor looks like this pixel, and a quarter of a window is plenty to average over. Thin structures are the same case stretched out, since a line one or two pixels wide is in the minority everywhere along its length.

Making the center the reference carries its own cost, which the formula below does nothing to hide. The center's value is itself a measurement, noise and all, so a badly corrupted pixel sets a badly chosen standard and holds on to its own error. Chapter 10.1, Bilateral filtering takes that up, including the repair of reading the weights from a pre-filtered copy.

The bilateral filter combines spatial distance with a difference in measured intensity (Tomasi & Manduchi 1998):

The spatial width $\sigma_s$ is measured in pixels; the range width $\sigma_r$ is measured in image-value units. An edge whose contrast is large compared with $\sigma_r$ contributes little weight across its boundary. In the limit $\sigma_r\to\infty$, all range weights approach one and the filter becomes Gaussian smoothing. Merely making $\sigma_r$ larger than the noise does not erase an edge with still larger contrast.

If $\sigma_r$ is too small relative to the noise, valid neighbors appear different and get rejected. On a constant patch with independent noise, $y_p-y_q$ has standard deviation $\sqrt2\sigma$ for $p\ne q$. This supplies a scale for selecting the range width, although its best setting also depends on the detail to preserve.

for each output pixel p:
    total = 0
    weight_sum = 0
    for q in spatial_window(p):
        w = exp(-distance_squared(p,q)/(2*spatial_width²))
        w *= exp(-(y[p]-y[q])²/(2*range_width²))
        total += w * y[q]
        weight_sum += w
    out[p] = total / weight_sum

The spatial weights can be tabulated once for the chosen window, since they are the same at every pixel; the range weights and their sum have to be evaluated per output neighborhood, because they depend on that neighborhood's values. Keep the input separate from the output so that every weight uses the same image. The self-weight is positive and prevents a zero denominator. For color images, a shared weight computed from a color-vector distance avoids choosing incompatible neighborhoods for different channels. Its metric must account for the chosen color space and noise scaling; Chapter 10.1, Bilateral filtering develops these choices and fast implementations.

The probe in Figure 13.3.4⛶ fullscreen shows how weights change at a boundary. Since noise affects the weights as well as the averaged values, the fixed-weight $N_{\mathrm{eff}}$ is a diagnostic of weight concentration, not an exact noise-reduction guarantee for the bilateral filter.

Strong or repeated filtering can flatten gradients into plateaus, retain isolated noisy pixels, or erase low-contrast texture. Similar edge-aware goals motivate anisotropic diffusion (Perona & Malik 1990), whose iterative update is a different operation.

13.3.4 Denoise color more than brightness🔗⧉

Vision generally resolves luminance detail more finely than chromatic detail. Many denoisers therefore smooth chroma more strongly. This is a tolerance, not permission to erase arbitrary color structure: isoluminant colored edges and fine saturated patterns can be visible.

A luminance/chrominance representation provides separate controls for brightness grain and color blotches. Stronger chroma smoothing can clean a neutral wall while retaining luminance texture; across colored boundaries it can bleed colors. RGB methods can also learn or implement such coupling. Choose the representation and weights for the image, rather than treating one channel layout as necessary for good denoising.

13.3.5 What each filter destroys🔗⧉

Compare results on flat regions, edges, and texture. Matching residual noise in a constant patch gives one useful operating point, but does not determine a universal ranking. An impulse-removal test and a Gaussian-noise test reward different behavior. Also inspect the removed signal $y-\hat x$: recognizable structures suggest lost detail, while their absence does not prove the estimate is correct.

Gaussian smoothing blurs boundaries. The median suppresses minority features along with outliers. The bilateral filter depends on contrasts being distinguishable from noise. These are consequences of the estimators, not parameters that can always be tuned away.

With every filter of this chapter on the table, the most useful exercise is to run them against the same noisy image and watch the trade-offs move. The lab below splits one image into four panes, the split following the mouse, each pane with its own filter, its own sliders and its own live score (Figure 13.3.7⛶ fullscreen). Put the Gaussian beside the bilateral and slide across an edge; set a pane to the amplified error against the clean original to see where each filter cheats, or to the difference of two panes to compare them directly.

The limitation of a small window motivates the next methods. A faint texture may have too little evidence locally to distinguish it from noise. Multiscale methods look for related structure across scales; non-local methods search for repeated patches farther away; learned denoisers use regularities acquired from other images. Each broadens the evidence available to the estimate, while introducing its own assumptions about which structures should agree.

13.3.6 The noise these filters were shown with, and the noise a sensor makes🔗⧉

Every figure in this chapter adds its own noise, and adds a simple kind: Gaussian of one width everywhere, or a fraction of samples replaced outright. That keeps the comparison between filters fair, since all of them see the same input, and it is the case the estimators are usually derived for. It is not what a sensor delivers.

A real measurement carries two noises with different causes. Photon arrivals are Poisson, so the variance of the signal grows with the signal itself and a bright region is noisier in absolute terms than a dark one, while being far less noisy relative to its own level. The electronics add a roughly Gaussian read noise of fixed width, which is what dominates once the light is low. Poisson plus Gaussian is the working model, and Chapter 13.1, Noise models, transforms, and benchmarks sets it out with its parameters and how to measure them on a camera.

Two things follow for the filters here. A width chosen for one part of the picture is wrong in another: the range width of a bilateral that keeps edges in the shadows will smooth texture away in the highlights, and a fixed threshold anywhere in this chapter inherits the same problem. And the domain matters, since these figures work on display-encoded values, where the encoding has already flattened much of the signal dependence that a raw file still carries.

There are two ways out, and they are worth separating because they fail differently. Either change the data so that one width is right everywhere, or leave the data alone and change the width as you move across the picture.

Filtering in a stabilized domain🔗⧉

A variance-stabilizing transform is a curve applied to the pixel values whose job is to make the noise the same size everywhere. For pure Poisson counts the classical choice is Anscombe's, $f(y)=2\sqrt{y+3/8}$, whose output has a variance close to one whatever the mean, so a bright region and a dark one arrive at the filter equally noisy. Raw sensor data is not pure Poisson, since read noise sits underneath it, so the useful version is the generalized transform built on the affine model $\operatorname{Var}=a\,x+b$ of Chapter 13.1, Noise models, transforms, and benchmarks:

The recipe is then the obvious one. Transform the raw values, run the bilateral with a single range width, since after the transform one width means the same thing in the shadows and the highlights, and transform back. In stabilized units the sensible setting is explicit rather than tuned: two pixels of equal true brightness differ by about $\sqrt2$ noise units, so a range width of two or three units keeps a genuine edge and admits the noise.

Three cautions come with it, and they are the reason this is not simply the right answer. The inverse is not the algebraic one: averaging happened in the transformed space, and because the transform is curved, mapping the average back with $f^{-1}$ returns a value biased low. Use the exact unbiased inverse (Mäkitalo & Foi 2013 (generalized Anscombe)), which is tabulated for this purpose. The stabilization itself degrades at very low counts, a few photons per pixel, which is exactly the regime where denoising matters most. And the model has to hold: subtract the black level first, keep clipped samples out, and do this in raw, before demosaicking and resampling correlate the noise and destroy the per-pixel Poisson story (Chapter 15.8, Joint demosaicking and denoising).

Or vary the widths instead🔗⧉

The other route leaves the measurements in their own units and makes the filter's parameters follow the noise. The range width becomes a function of position,

with $k$ around two or three, so the kernel always spans a couple of noise widths and no more. In the shadows that is a narrow window, which is what protects the little contrast left there; in the highlights it opens up, because a difference that would be a real edge in the shadows is noise on a bright wall.

The estimate $\hat I_p$ is the subtle part. It cannot be the pixel's own measured value, or the parameter that is supposed to reject the noise is being set by that noise, and a bright noise spike both raises its own threshold and rejects its neighbors. Use a pilot estimate: a coarse pre-filtered copy, a downsampled version, or the output of a first pass. This is the same instinct as reading the range weights from a median-filtered guide (Chapter 10.1, Bilateral filtering), and it has the same cost, that the pilot's mistakes become the filter's parameters.

Two practical notes. Scaling the width by the center pixel's own noise level makes the weight asymmetric, $w(p,q)\neq w(q,p)$, which is harmless for a normalized average but matters if the weights are later reused as a graph affinity, where a symmetric form such as the average of the two levels is the safer choice. And after white balance the per-channel gains have scaled the noise differently in each channel, so one width across red, green and blue is wrong by exactly those gains: either apply the widths before the gains, or scale them with the gains.

The two routes meet in the middle. Scaling the range width by the local noise is, to first order, what filtering in the stabilized domain does, since the transform's slope at a value is precisely one over the noise there. The stabilized route is exact for its assumed model and comes with a bias to undo; the varying-width route never touches the data, needs no inverse, and inherits whatever the pilot estimate gets wrong. Either way, test the result the same way: flat patches at several brightness levels and an edge down in the shadows, where a filter tuned on a midtone chart is at its worst. Chapter 13.1, Noise models, transforms, and benchmarks carries the benchmarks that say whether any of it helped.