BookNotesSlidesP-sets Syllabusdraft · v0.2.49.1
◀▲⌂▶
💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.

24.2 Mixed-lighting white balance⧉

Photograph a room at dusk with a lamp on and the curtains open, and the camera cannot solve the problem with a single global adjustment. The lamp throws a warm, orange light on one side of a face; the window lets in cool, bluish daylight on the other. White-balance for the lamp and the daylit cheek turns blue; white-balance for the window and the lamp-lit cheek turns orange. There is no global gain that neutralizes both, because the right correction is different in different parts of the frame, and an ordinary white balance has exactly one correction to give.

This is where the global white balance of Chapter 8.10, Auto-exposure and auto white balance fails. There, the camera's job was to estimate one illuminant and divide it out with a von Kries diagonal gain $k_R, k_G, k_B$: a single multiply in linear light. That model assumes one light color over the whole scene. Mixed lighting violates the assumption at the root: the illuminant is no longer one color but a field of colors that varies across the image. So we cannot estimate one gain; we have to estimate, at every pixel, which light is doing the lighting there, and correct accordingly.

[figure fig-mixed-illuminant-scene not built]
Figure 24.2.1. Why one white balance is not enough. A scene lit by two differently-colored sources (warm interior light and cool daylight through a window) cannot be neutralized by a single global gain: balancing for one light leaves the other cast. A correct fix needs spatially-varying, multi-illuminant white balance.

24.2.1 Why a global gain must fail⧉

The argument is short and simple, and it tells us exactly what the right unknown is. Recall the white-balance model: under one illuminant of color $L = (L_R, L_G, L_B)$, a surface of reflectance $\rho$ records as $\rho \cdot L$ in each channel, and white balance divides by $L$ (three gains $k_c = 1/L_c$) to recover the reflectance. The whole method rests on $L$ being the same everywhere.

Now put two lights in the room, a warm one $L^{(1)}$ and a cool one $L^{(2)}$. At a pixel $x$ the surface is lit by some blend of the two, depending on where it sits and what it faces: the cheek by the lamp sees mostly $L^{(1)}$, the cheek by the window mostly $L^{(2)}$, and a point on the table between them sees a bit of each. Write the local light as a convex mixture

$$ L(x) = \alpha(x)\, L^{(1)} + \big(1 - \alpha(x)\big)\, L^{(2)}, \qquad \alpha(x) \in [0, 1], $$

where $\alpha(x)$ is the warm light's fraction after both endpoint colors have been normalized by the same convention. Overall illumination strength is a separate scalar $s(x)$: the linear pixel is approximately $C(x)=s(x)\rho(x)L(x)$, with channelwise multiplication. Below we absorb $s$ into the effective surface color $\rho$; white balance removes a color cast, not shading. Dividing by $L(x)$ requires a different gain at each pixel. A constant $\alpha$ permits a global correction even with two lights, provided their relative contributions are the same everywhere. When that ratio varies, we need a mixing map. It often varies smoothly across a surface, but a cast shadow can make it change sharply.

24.2.2 Estimating the per-pixel mixture: an under-determined inverse problem⧉

Here is the same shape we have seen all part. We want to recover, at each pixel, both the surface reflectance $\rho(x)$ (three numbers) and the mixing coefficient $\alpha(x)$ (one number), four unknowns, from the recorded color $\rho(x)\,L(x)$, three numbers. More unknowns than measurements per pixel. The model is under-determined as written: at any single pixel, a redder reading could mean a redder surface or more of the warm light, and the data alone cannot tell which (Figure 24.2.2). We cannot solve it pixel by pixel; we need a prior.

This is another recurrence of L7.8: the prior is not optional; see Chapter 14.1, What can super-resolution actually do?. Mixed-lighting white balance is a clean instance: the per-pixel illumination split is ambiguous from one pixel's color, and assumptions about both material colors and illumination are needed to select a solution.

The two light colors. Before solving for the map, we need the two endpoints $L^{(1)}$ and $L^{(2)}$ of the mixture. Hsu et al. take them as known or estimated up front: picked by the user from neutral regions clearly under each light, or estimated from the image. Tungsten plus daylight is a common case. Color temperature alone does not specify an arbitrary source spectrum or its camera RGB; a neutral reference under each light provides more direct evidence. With the endpoints fixed, the illumination chromaticity has one free mixing coefficient. The surface color is still unknown.

Material and illumination priors. Illumination is often smoother than surface texture, but smoothness alone does not resolve this problem. For any proposed mixing map with nonzero channel values, choosing $\rho(x)=C(x)/L(\alpha(x))$ fits the data exactly. Hsu et al. also exploit a limited palette of material colors. A schematic joint objective makes the missing constraint explicit:

$$ (\hat\alpha,\hat\rho)=\arg\min_{\alpha\in[0,1],\rho}\sum_x\lVert C(x)-\rho(x)L(\alpha(x))\rVert^2+\lambda\sum_x w(x)\lVert\nabla\alpha(x)\rVert^2+\mu\Phi(\rho). $$

Here $\Phi$ constrains the surface-color explanation, for example through a material palette or known neutral regions; $\lambda$ and $\mu$ set the penalty strengths. This illustrates the required constraints rather than reproducing Hsu et al.'s exact solver. $C(x)$ is the recorded color and $w(x)$ an edge-aware weight that relaxes the smoothness penalty across genuine image boundaries so the mixing map can change where the scene does, exactly the affinity / edge-preserving machinery of Part 10, Edge-preserving techniques (and a sibling of the guided refinement that snaps the transmission map to edges in Chapter 24.3, Dehazing). Given such material constraints, the smoothness term helps propagate estimates: it propagates the few confident estimates (neutral or near-neutral pixels, where reflectance is colorless and the color reads off the light directly) across the textured, ambiguous regions, filling in a coherent mixing map where no single pixel could decide for itself.

fig-mixed-wb-mixing-map
Figure 24.2.2. The per-pixel mixing map. Left: a scene lit by a warm lamp and cool window at once, the cast sliding from orange to blue across the frame. Right: the recovered mixing coefficient $\alpha(x)\in[0,1]$, a smoothly varying map of how much of the warm light is in the blend at each pixel (warm/orange where the lamp dominates, cool/blue where the window does, a soft transition between). The map is the unknown a single global gain cannot represent; a smoothness prior is what makes it recoverable from one image. (Illustration: a hand-built mixing field, not a faithful algorithm output.)

24.2.3 Correcting each region for its own light⧉

With the mixing map $\hat\alpha(x)$ in hand, the correction is mechanical: at every pixel reconstruct the local light $L(x) = \hat\alpha(x)\,L^{(1)} + (1 - \hat\alpha(x))\,L^{(2)}$ and divide the pixel by it, a per-pixel von Kries gain instead of one global gain. Where the lamp dominates we divide out the warm light; where the window dominates we divide out the cool light; in between we divide out the blend. With accurate estimates and a suitable diagonal model, neutral surfaces in both regions lose their casts at once, the thing a single gain could never do. The art and craft connection is direct: this is what a careful colorist does by hand with local color correction, masking the warm and cool regions and grading each separately. The method does it automatically and continuously, the per-pixel analog of dodging-and-burning the color of the light.

The estimate also supports lighting edits. Replacing $L(x)$ changes the local light color. To simulate switching off a source, however, retain its unnormalized contribution: write $C=\rho(i_1L^{(1)}+i_2L^{(2)})$ and set the appropriate amplitude $i_1$ or $i_2$ to zero. Changing only the normalized mixture does not reproduce the brightness loss. Neither operation predicts new shadow geometry or view-dependent reflections.

Video 24.2.1. Video: Light Mixture Estimation for Spatially Varying White Balance: estimating, at every pixel, the mixture of two illuminants and white-balancing each region for the light that actually falls on it. Hsu, Mertens, Paris, Avidan & Durand 2008 (Hsu et al. 2008).
fig-light-mixture-wb-1
Figure 24.2.3. Mixed lighting, where a single white balance must lose. The scene is lit by two illuminants of different color at once (a warm indoor source and cool daylight), so the cast a camera should remove changes from one part of the frame to the next. Balance for the warm light and the daylit regions go blue; balance for the daylight and the warm regions go orange; there is no one gain that neutralizes both, which is the whole problem a global white balance cannot solve Hsu et al. 2008.
fig-light-mixture-wb-2
Figure 24.2.4. The spatially-varying fix, in overview. Rather than guess one illuminant for the frame, the method estimates at every pixel how much of each of the two lights is contributing (a per-pixel mixture of the two illuminants) and then white-balances each region for the light that actually falls on it. The reframing to keep is that a mixed-light scene has no single answer, so the unknown becomes a smoothly-varying mixing map rather than a single gain Hsu et al. 2008.
fig-global-vs-spatial-wb
Figure 24.2.5. Global versus spatially-varying white balance. The same mixed-lit scene corrected four ways: the input; a single global gain balanced for the lamp (the window side goes blue); a single global gain balanced for the window (the lamp side goes orange); and the spatially-varying multi-illuminant result, which carries the right correction into each region so warm-lit and daylit areas read neutral at once. A global gain can only ever neutralize one of the two lights; treating the illuminant as a per-pixel mixture neutralizes both. (Illustration: a hand-built scene, not a faithful algorithm output.)
fig-light-mixture-wb-8
Figure 24.2.6. Global versus spatially-varying white balance, on a real result. The same mixed-light scene is corrected two ways: a single best global gain, which can only ever neutralize one of the two lights and leaves the other tinted, and the multi-illuminant result, which carries the right correction into each region so warm-lit and daylit areas both read neutral at once. This is the gain from treating the illuminant as a per-pixel mixture rather than one number for the frame Hsu et al. 2008.

Estimate a spatial illuminant map before applying local white balance. Check neutral surfaces under both lights and inspect transitions at shadows. Material assumptions, endpoint errors, and the diagonal color model limit the correction; smoothness by itself cannot identify the light.

24.2.4 What came after⧉

Hsu et al. used known or estimated endpoint colors and a material-color model. Later work pushed on every part of that: removing the user, allowing more than two lights, learning the spatially-varying correction, and building the datasets to train and measure it.

The pattern: Hsu's estimate the mixture, then correct each region is the conceptual core, and everything after either removes a hand-set assumption (the seeded light colors, the two-light limit), replaces the smoothness prior with a learned one, or builds the data to do so. It also connects outward to intrinsic-image / illumination decomposition (separating a photo into reflectance and shading/illumination) of which spatially-varying white balance is the color-of-the-light special case.