BookNotesSlidesP-sets Syllabusdraft · v0.2.49.1
◀▲⌂▶
💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.

8.10 Auto-exposure and auto white balance⧉

Auto-exposure chooses capture settings and auto white balance estimates a color correction. Both depend on uncertain scene information: a dark object and an underexposed bright object may yield the same measurement, as may a colored surface and colored illumination.

8.10.1 Auto-exposure: metering⧉

A meter estimates scene brightness, then the exposure program chooses shutter duration, aperture, and ISO within constraints. Shutter duration and aperture affect photons collected; ISO affects gain and often the selected exposure. A motion limit, depth-of-field requirement, or highlight constraint can take priority over average brightness.

Average and center-weighted metering. The simplest meter averages the whole frame to one number and exposes so that average is middle gray (see Chapter 5.2, Exposure and capture timing). It is fooled by anything that is not average: a bright sky pulls the average up, so the meter under-exposes and the foreground falls into shadow (the classic backlit-subject silhouette); a snow field reads too bright and is rendered a dingy gray. Center-weighted metering improves the guess by weighting the middle of the frame most, on the assumption that the subject is usually near the center.

Matrix / evaluative metering. Combine measurements from several regions, optionally using focus, face, color, or distance information. Different cameras use different decision rules. Nikon's 3D Color Matrix metering, for example, compares zone brightness and color and the subject distance reported by the lens against a database of reference scenes, which Nikon describes as tens of thousands of photographs; Canon's evaluative metering and other makers' equivalents are cousins. A subject-weighted result may deliberately sacrifice some highlights; an image-wide average does not establish the photographer's intent.

Spot metering is the opposite extreme: meter one small zone and place that tone at middle gray. It hands the photographer exact control over which tone is anchored (the basis of Ansel Adams's Zone System) at the cost of having to choose wisely.

fig-ae-metering
Figure 8.10.1. Auto-exposure metering modes (interactive). Pick a scene and a metering mode and the demo scales the exposure (a multiply in linear light) so the metered region averages to ≈18% gray, reporting the metered tone, the gain, and the fraction of highlights it clips. Average meters the whole frame: a bright sky or water drags the average up, the gain comes out small, and the subject falls dark. Center-weighted biases toward the middle but is still pulled by a big bright surround. Spot meters one small zone, moved and resized with the sliders: it places that zone at middle gray at the cost of clipping the bright background. That is correct only for a subject of average reflectance; spot-meter a white shirt and it comes out gray. Same scene, same target: only where you meter changes, and the exposure with it. Try the backlit sunset scene, then move the spot from the dark foreground to the bright sky.

Learned metering. An estimator can learn exposure preferences or use recognized subjects to weight the measurement. Burst pipelines can choose shorter frames to limit motion and saturation, then merge them to reduce noise. Exposure compensation and AE lock remain useful when the intended subject or mood differs from the automatic choice.

For video and preview, both exposure and white-balance estimates also need temporal stability. Damping small frame-to-frame changes reduces flicker, but too much damping delays adaptation after a lighting change.

A zone pattern retains information that an overall average discards. A bright upper region, a darker lower region, and a focused subject between them can suggest backlighting, whereas a uniformly bright frame may be snow or a white wall. Such evidence can guide an exposure decision, but does not settle whether the photographer wants a silhouette. Exposure compensation supplies that missing instruction. The progression from one average to spatial patterns and learned statistics adds evidence about intent without making intent observable directly.

8.10.2 White balance and color constancy⧉

White balance is the harder color-estimation problem. For a simple diffuse surface, the spectrum reaching the sensor depends on the product of the illuminant spectrum and the surface reflectance, so a white shirt under tungsten light sends warm, orange-ish light to the sensor, and the very same shirt under open shade sends cool, bluish light. Yet to a human the shirt looks white under both, because the visual system discounts the illuminant, the color constancy of the perception chapter. A camera has no such automatic constancy; it must estimate the illuminant and divide it out, and that operation is white balance (Figure 8.10.2).

fig-wb-before-after
Figure 8.10.2. White balance, before and after, on two real captures. Top, an arcade under warm artificial light; bottom, a white fence in open shade. Each is shown first at the camera's daylight balance, which is what a daylight film would have recorded, and then balanced on a white surface in the scene. Before balancing, the warm scene's floor tiles read 1.53 : 1 : 0.51 in red, green and blue, and the shaded fence 0.89 : 1 : 1.16; after it, both read neutral. White balance is the camera's stand-in for the eye's color constancy.

A common approximation uses independent gains in a suitable linear color basis Von Kries:

$$ R' = k_R\,R, \qquad G' = k_G\,G, \qquad B' = k_B\,B, $$

These gains form a diagonal transform. A known neutral supplies their ratios, with a common scale left for exposure. Camera channels and cone-like adaptation coordinates are different bases; Bradford adaptation is not simply a diagonal scaling in arbitrary RGB. See Chapter 4.2, White balance and white point for that distinction and Land & McCann 1971 for the Retinex connection.

Subtract the calibrated black level before applying sensor-channel gains. Apply them before nonlinear rendering, in the linear basis for which they were estimated. A common scalar exposure commutes with a linear color matrix, but unequal channel gains generally do not: $MD\ne DM$. A gain moved after encoding needs the conjugate operation $g(Dg^{-1}(v))$, not the same numerical multiply. White balance may precede demosaicking; its placement must match the reconstruction and color profile.

8.10.3 White balance on film⧉

Before sensors, white balance was chosen when the film was bought. A color film's three emulsion layers are balanced so that one particular illuminant records as neutral. Daylight film is balanced for about 5500 K, the mix of sun and sky at midday; tungsten film, sold as Type B, for about 3200 K, the color of studio lamps (Figure 8.10.3). Shoot daylight film under household or studio lamps and everything comes out orange, which is the "before" panel of Figure 8.10.2; shoot tungsten film outdoors and everything comes out blue.

fig-film-by-illuminant
Figure 8.10.3. White balance you bought in a box. Left, a tungsten-balanced slide film, Kodak Ektachrome 64T, balanced for studio lamps at about 3200 K. Right, a daylight-balanced print film, Kodak Gold 200, balanced for about 5500 K. The choice of roll fixed how every frame on it would render the color of the light. (Photographs CC BY-SA; the figure is CC BY-SA 4.0.)

A photographer caught with the wrong film under the wrong light used a conversion filter on the lens. A blue 80A filter lets daylight film record correctly under 3200 K lamps, at the cost of about two stops of light, because it works by absorbing much of the red. An amber 85B lets tungsten film be used in daylight, at the cost of about two thirds of a stop. Photographers reasoned about these filters in mireds, the reciprocal of color temperature scaled by a million,

$$\text{mired} = \frac{10^6}{T},$$

because a given filter shifts a source by roughly the same number of mireds whatever its temperature. Going from 3200 K (about 312 mireds) to 5500 K (about 182) is a shift of about −131 mireds, which is the 80A's rating, and the 85B is the reverse. Fluorescent tubes, whose spectra are spiky rather than smooth, called for their own filters and often a magenta color-compensating filter to cancel their green cast.

The optical filter is the film-era version of the per-channel gains above, with two differences. It acts on the spectrum before the film records it, so it is a real change to the light rather than a rescaling of recorded numbers, and it costs exposure, which a digital gain does not (though a gain amplifies noise in the channel it raises). And it applies to the whole frame, so a scene lit by two sources, a lamp-lit room with daylight at the window, could not be corrected by any filter on the lens. Film and cinema crews instead filtered the lights themselves, with orange gels on windows or blue gels on lamps, which is the problem Chapter 24.2, Mixed-lighting white balance solves in software.

The two kinds of film also differed in how forgiving they were. A slide is the final image, so a color cast on reversal film is there for good. A color negative is printed, and the printing lab set color filtration for each negative, often with an automatic analyzer, which removed much of a cast after the fact. That printing step was an automatic white balance, run on the negative rather than on sensor data. A digital camera does the same thing per frame, and a raw file defers it entirely, so the choice made at the film counter is now a slider in the editor (Chapter 5.5, Developing and editing a photograph).

8.10.4 Automatic white balance⧉

In practice the camera has no labeled neutral patch, so it must guess the illuminant from the image itself: automatic white balance (AWB). The classic assumptions are statistical. Gray-world Buchsbaum 1980 assumes the scene averages to gray, so the channel gains are

$$ k_c = \frac{\text{mean gray}}{\text{mean}_c}, $$

Here mean gray can be the average of the three channel means. Use only valid samples and guard near-zero channel means; normalizing the gains to a chosen reference channel separates their common exposure scale. This scales the channels so their means agree (Figure 8.10.4). The bright-pixel (white-patch) assumption (Lee 1986) instead takes the brightest pixels to be a white highlight reflecting the illuminant directly, and balances on those. Both fail in predictable ways: gray-world tips on a scene dominated by one color (a forest, a red wall), bright-pixel tips on a colored specular highlight.

Reject saturated samples and unstable ratios near black when estimating gains. White-patch statistics can select different surfaces in different channels, and a scene need not contain a neutral highlight. Under mixed lighting, one global gain cannot neutralize every region. A neutral reference helps, but white balance cannot reconstruct wavelengths missing from the illumination.

fig-awb-methods
Figure 8.10.4. Four simple estimators on the photographs of Figure 4.2.1 (interactive). Each photograph is relit under a known light, so every estimate can be scored as the angle between the estimated and the true illuminant in RGB, the usual measure in color constancy. Gray world takes the channel means; brightest pixels averages the top few percent by luminance, a white patch that tolerates noise; shades of gray (Finlayson and Trezzi, 2004) replaces the mean with a Minkowski p-norm, which is gray world at p = 1 and tends to the per-channel maximum as p grows; gray edge (van de Weijer, Gevers and Gijsenij, 2007) asks the same of the image's derivatives, on the assumption that edges, not surfaces, average to gray. Saturated pixels and pixels too dark to carry a ratio are left out. Over the fifteen frames at the defaults (brightest 2%, p = 6) the mean errors are about 8° for gray world, 4.8° for brightest pixels, 5.4° for shades of gray and 4.4° for gray edge, against 34.5° for leaving a tungsten frame uncorrected. Each fails where its assumption does: gray world is 21° off on a yellow café front and 14° on a warm still life, gray edge 11° on a portrait against smooth walls, and on the two frames shot under daylight every method makes the picture worse, since there was no cast to remove. Interactive: choose the photograph and the estimator, set the brightest-pixel percentage and the Minkowski p, and mark the pixels that vote; the plot shows every estimate against the true light in rg chromaticity.

Learned estimators can use richer image statistics than gray-world. FFCC Barron and Tsai, 2017 estimates illumination from a small periodic log-chroma histogram.

For positive linear values, let $u=\log(G/R)$ and $v=\log(G/B)$. Under a global diagonal gain, $u'=u+\log(k_G/k_R)$ and $v'=v+\log(k_G/k_B)$. Thus the model turns illumination into a histogram translation. FFCC uses Fourier-domain filtering, resolves periodic ambiguity, and estimates a posterior rather than only a peak. The periodicity belongs to its histogram representation, not to physical illumination.

fig-ffcc-pipeline
Figure 8.10.5. The Fast Fourier Color Constancy pipeline, from Barron. The input image is binned into a two-dimensional log-chroma histogram; a learned filter is convolved over that histogram (efficiently, in the Fourier domain) to produce a heat map whose peak localizes the illuminant; and that estimate is read back out as the per-channel gains that white-balance the image. The reframing to keep is that color constancy has become localization in a histogram: because changing the light merely translates the log-chroma histogram, finding the light is finding the shift. This is the method's overview (Barron 2017).
fig-ffcc-model
Figure 8.10.6. The FFCC model, from Barron. The estimator is a single learned filter applied to the log-chroma histogram, trained discriminatively so that convolving it with any scene's histogram puts its response peak at the true illuminant. Because illuminant change is a translation of the histogram, one shift-invariant filter (a convolution) suffices, and the whole estimate reduces to a few small Fourier-domain operations, which is what makes the method both accurate and exceptionally fast. This is the learned model at the heart of the method (Barron 2017).
fig-ffcc-localization
Figure 8.10.7. Illuminant estimation as localization, from Barron. Convolving the learned filter with a scene's log-chroma histogram yields a response surface over the chroma plane; the white balance is read off as the location of its peak. Seeing the estimate as the argmax of a heat map is what makes the rest possible: it turns "what color was the light?" into "where is the peak in this histogram response?", a question a convolution answers directly, and one whose confidence we can read from how sharp the bump is. From the FFCC method (Barron 2017).
fig-ffcc-von-mises
Figure 8.10.8. Reading a confidence from the FFCC response, from Barron. Rather than just take the peak, the method fits a von Mises distribution (the natural bell curve for a quantity that wraps around, as hue does) to the response surface. The fitted mean gives the illuminant estimate and its spread gives a calibrated uncertainty, so the camera knows not only its best guess but how much to trust it, which matters when fusing the estimate over a video or with other cues. The lesson is that localizing in a histogram hands you a full distribution, not just a point. From the FFCC method (Barron 2017).
fig-ffcc-result-1
Figure 8.10.9. FFCC white-balance results on the Gehler–Shi benchmark, from Barron. Scenes from the standard color-constancy dataset, each shown with its raw cast and after correction by the learned illuminant estimate, with the recovered light compared to the ColorChecker ground truth. Notice how a method whose entire inference is a small Fourier-domain convolution over a histogram nevertheless tracks the true illuminant across very different scenes: the priors gray-world only gestures at have been learned. The first set of published results (Barron 2017).
fig-ffcc-result-2
Figure 8.10.10. Further FFCC results on Gehler–Shi, from Barron. A second set of benchmark scenes corrected by the method, including cases that defeat the classic statistical assumptions: frames dominated by a single strong color, where gray-world would drain the very hue that belongs in the scene. The takeaway is that the learned histogram filter has absorbed enough of natural-scene statistics to keep the color when gray-world would lose it, which is exactly the gap between a hand-coded statistic and a learned estimator that this chapter has been tracing. More published results (Barron 2017).
Skin tones: a calibration target, a memory color, and a fairness caveat

Test exposure and white balance across a range of skin tones and lighting conditions. A face detector should not imply that every face belongs at one brightness or one color. Historical calibration practices, including Shirley cards, motivate the broader discussion in Chapter 40.2, Ethics of computational photography.

In CompPhotoRoom

In Chapter 8.13, A raw developer in the browser, White balance is a real Kelvin axis (Planck's law against the CIE observer), and the Auto button on the Tone panel is a learned auto-adjustment: a regression trained on the MIT-Adobe FiveK dataset (Bychkovsky et al. 2011) against expert C, in the app's own slider space. Its ? tells the full story.

This is a localization problem in a representation chosen to simplify the physics. The histogram retains the distribution of channel ratios while discarding their positions in the picture. A learned filter can score possible shifts using distributions seen in training. The estimated two-dimensional shift specifies relative channel gains, leaving their common scale to exposure. The method replaces the single gray-world assumption with a learned statistical model, but surface color and illumination can still admit competing explanations.