24.1 Illumination related effects in a single image🔗⧉

A single photograph mixes things the eye effortlessly keeps apart. Look at a white shirt half in shadow and you read it instantly as white-and-shadowed (one surface, unevenly lit) not as a gray shirt that happens to be pale at the top. The camera cannot do this. It records only the product: at every pixel, the light that fell on the surface times how much of that light the surface sent back. Write it as $I = R\cdot S$: illumination (shading) $S$ times reflectance $R$ L2.6show. The shirt's whiteness lives in $R$; the shadow lives in $S$; under this simplified diffuse model the camera hands you their product and leaves you to guess which is which.

This chapter is the project of un-mixing that product from one image: pulling the light apart from the surface, or one layer of light apart from another. And it is the same story as super-resolution and deblurring told in a new costume: the un-mixing is under-determined, so a prior is what makes recovery possible L7.8show. The unifying frame is the intrinsic-image decomposition, and everything else in the chapter (white balancing a scene lit by two different lamps, deleting a reflection off a shop window, lifting a shadow off a face, killing the glossy highlight on a leaf) is a special case of it: I care about one layer, and I have a cue that pins it down.

A standing note on encoding, since every method here depends on it L4.6show. The relation $I = R\cdot S$ is multiplicative, so the natural place to work is log, where it becomes additive, $\log I = \log R + \log S$: a multiplicative shading or illuminant factor becomes a clean offset you can manipulate with gradient and affinity tools. The layer-superposition effects (a glass reflection summed onto a transmission, a specular term added onto the diffuse) are instead genuinely additive in radiance, so those must be separated in linear light, before gamma. Declaring the space up front is not pedantry: do reflection removal in gamma and the "sum" you are trying to un-sum is not actually a sum.

24.1.1 Intrinsic images: the unifying frame ($I = R\cdot S$)🔗⧉

Start with the decomposition that names everything else. Split one photograph into two intrinsic layers: a reflectance image $R$ (the surface's own albedo, the color and lightness of the paint, independent of how it happens to be lit) and a shading image $S$ (the illumination, the play of light and shadow across the geometry, treated as independent of local albedo in the simplified direct-illumination Lambertian model; interreflection and non-Lambertian materials complicate this separation). Per color channel, $I = R\cdot S$; in log it is additive, $\log I = \log R + \log S$. The reflectance layer looks like a flat, evenly-lit version of the scene with all the modeling light removed; the shading layer looks like a smooth gray rendering of the geometry with all the texture and markings gone (Figure 24.1.1⛶ fullscreen).

Why is this the umbrella? Because once you can say "this variation is light and that variation is surface," every other effect in the chapter becomes an edit on one layer and not the other. White balance corrects the color of $S$. Shadow removal flattens a dark region of $S$ while leaving $R$ alone. Highlight removal strips a specular term that has contaminated $S$ in the wrong color. Reflection removal is the one outlier (it un-sums two whole images rather than splitting one into factors) but it is the same un-mixing instinct. Name the layers and the rest is bookkeeping.

And why is it hard? For the reason in the big-lesson box: infinitely many pairs $(R,S)$ multiply to the very same $I$. Halve the reflectance everywhere and double the shading and the product is unchanged; the data alone cannot prefer one factorization over another. You need a prior that says which kinds of variation are reflectance and which are shading.

The classic prior is Retinex (Land & McCann 1971 Land & McCann 1971), and its intuition is one sentence: reflectance changes are sharp, shading changes are smooth. A painted edge (where one surface color abuts another) produces a sudden jump in the image. A shading change (light falling off across a curved surface, or the soft edge of a shadow) produces a gentle ramp. So walk along the image and look at the gradient of the log image: where it jumps hard, that variation is surface, so assign it to $\nabla R$; where it ramps gently, that variation is light, so assign it to $\nabla S$. The decision rule is a simple threshold on the gradient magnitude, $|\nabla \log I| > \tau$ for reflectance (Figure 24.1.2⛶ fullscreen). Having sorted every gradient into one of two piles, you integrate each pile back into an image by exactly the gradient-domain reconstruction of Chapter 10.8, Poisson image editing: solve the Poisson equation whose guidance field is the reflectance gradients, and again for the shading gradients L10.4show. Retinex is crude (a hard threshold mislabels soft material edges and sharp shadow edges, the two failure modes the rest of the field spends its energy on) but it captures the whole instinct in one move: edges are surface, ramps are light.

The modern version keeps the same recipe, data-fit plus a hand-built prior, recovered by maximum a posteriori (MAP) estimation, but enriches the priors and adds a third unknown. Barron and Malik's shape, illumination, and reflectance from shading (SIRFS, 2015) Barron & Malik 2015 recovers all three jointly from a single image: a reflectance prior that albedo is piecewise-constant and draws from a small palette; a shape prior that surfaces are smooth and bend gently; and an illumination prior that lighting is low-frequency (a few soft sources, not a thicket of hard ones). Maximizing the posterior under these priors is the same data-fit-plus-prior optimization that runs through the whole part. SIRFS is the high-water mark of hand-built priors for this problem; the learned successors, convolutional neural networks (CNNs) trained to predict $R$ and $S$ directly from a photo, live in Chapter 7.3, Deep learning, and they are the canonical instance of L7.6show (a learned operator swapping a hand-designed prior for one fit to data).

The benefit of factoring out the light is that you can put a different light back. Relighting is the intrinsic move pointed at faces: recover (often implicitly) the subject's shape and reflectance, then re-render under new illumination. Single-image portrait relighting (Sun et al. 2019) trains a network on a light-stage corpus (one person photographed under every lighting direction in turn) so that, given a single ordinary phone photo, it can re-illuminate the face to match any target environment map; Google's Portrait Light ships a gentler version, dropping in a repositionable virtual off-camera light whose direction and intensity are auto-chosen to complement whatever lighting the photo already had. It is the same separate-the-factors-then-recombine recipe as intrinsic images, with the light stage standing in for the prior; the multi-image light-stage capture itself is Part 35, Computational illumination.

Carry this forward: intrinsic decomposition is the generalization. The remaining sections each pick one layer to recover and bring a sharper cue than Retinex's blunt threshold to pin it down.

24.1.2 Multiple-light / spatially-varying white balance🔗⧉

Ordinary white balance, as set up in Chapter 4.1, Color space 101 and the ISP chapter, estimates one illuminant color for the whole frame and divides it out (L2.6show: white balance is a division). This is a useful diagonal approximation under spatially uniform illumination; it is not an exact spectral correction for every material. Mixed illumination adds a spatial problem to those colorimetric limits.

And real scenes are routinely not. Step into a room at dusk: warm tungsten lamps on one side, cool blue daylight through the window on the other. Or fire a flash indoors and let the warm ambient fill the rest. Or shoot a face half in direct sun (warm) and half in open shade (lit by the blue sky). A single global correction can neutralize one of these and only one: fix the tungsten side and the window goes glacial blue; neutralize the window and the lamp-lit side turns orange (Figure 24.1.3⛶ fullscreen). There is no one number that is right everywhere, because there is no one illuminant.

The fix is to let the white balance vary across the frame. Hsu and colleagues (2008) Hsu et al. 2008 model each pixel as lit by a blend of two known illuminant colors, $L_1$ and $L_2$, in proportions that vary spatially:

The unknown is the per-pixel mixing map $\alpha(x)$: how much of each light reaches that pixel. Solve for $\alpha(x)$ and you have a spatially-varying white balance: at each pixel you know the local illuminant color, so you divide that out and recover a neutral $R(x)$. The two endpoint colors $L_1,L_2$ come from the user clicking a warm and a cool region, from suitable unclipped dielectric specular highlights, whose color can constrain the illuminant, or from clustering the image's colors into two illuminant directions.

What makes this an intrinsic-image method, and not a new trick, is that it is recovering the color of the shading layer as it varies across the scene, a chrominance-only intrinsic split. The method also uses material-color structure, including a limited palette of repeated reflectances, to constrain the mixture. Spatial regularity helps where lighting varies smoothly, but cast shadows can create abrupt mixture changes. That smoothness is what regularizes the otherwise-underdetermined per-pixel estimate, the same "ramps are light" assumption, now applied to the color of the illumination rather than its brightness. As always, do the dividing in linear light, where mixing and removing illuminants is the operation the radiometry says it is L4.6show.

The easier multi-image version adds an independent illumination measurement: a flash/no-flash pair gives you one frame under a light of known color (the flash) and one under the ambient, so registered linear differencing can isolate a flash contribution where the ambient stayed constant; motion, flash shadows, and specular changes remain, that method lives in Part 35, Computational illumination. The learned single-image version (a network that predicts a per-pixel illuminant map) is in Chapter 7.3, Deep learning.

24.1.3 Reflection removal — pulling apart a transmission and a glass reflection🔗⧉

Photograph a painting behind glass, or a street through a shop window, and you capture two scenes summed on top of each other: the transmitted layer $T$ you meant to shoot, plus a reflection $R_{\text{refl}}$ of whatever was behind the camera. This effect is additive, unlike the multiplicative $R\cdot S$: the glass passes one image and mirrors another, and the sensor adds them:

Recovering both $T$ and $R_{\text{refl}}$ from a single $I$ is hopelessly under-determined: two unknown images, one observed sum, with no constraint tying them down. This is the purest prior territory in the chapter L7.8show: the data constrains the sum but does not uniquely determine the split. Priors or additional measurements select among the compatible layers. Reflection removal is useful here because several different priors have been proposed for it, each exploiting a different physical cue that the reflection differs from the transmission in some measurable way (Figure 24.1.4⛶ fullscreen).

• Defocus / blur. You focus the camera on the transmitted scene, so the reflection (typically at a different distance) falls out of focus: it contributes weak, broad gradients, while the transmission contributes sharp ones. Li and Brown's relative-smoothness method (2014) exploits this kind of difference: assign the sharp gradients to $T$ and the soft ones to $R_{\text{refl}}$. Note that this is Retinex again, run on focus instead of on light-versus-surface, the same "sort the gradients, then integrate" move from the intrinsic-image section.

• Ghosting. A thick glass pane reflects light off both its front and back surfaces, so the reflection arrives twice, slightly shifted, a doubled ghost. That shift can be modeled by a two-impulse kernel whose displacement and relative weights must be estimated; spotting the characteristic doubling lets you identify the reflected layer and subtract it.

• Sparse-gradient prior. Levin and Weiss (2007) Levin & Weiss 2007 lean on natural-image statistics: each layer on its own has sparse, heavy-tailed gradients (mostly flat, a few strong edges), so their sum has too many edges: its gradient histogram is too dense. Among all the ways to split $I$ into $T+R_{\text{refl}}$, prefer the one that minimizes the total number of edges across both layers, optionally guided by a few user scribbles marking which layer an edge belongs to. This is the very same sparse-gradient prior that breaks the chicken-and-egg of blind deblurring, the natural-image prior earning its keep again.

• Polarization. Light reflected off glass at a glancing angle is partially polarized (the Brewster effect), while the transmitted light is much less so. A polarizing filter therefore attenuates $R_{\text{refl}}$ differently from $T$. The genuine method rotates a real polarizer and takes two frames (that lives in Part 35, Computational illumination) while a polarization sensor can measure several analyzer directions in one exposure. A conventional single RGB image does not directly measure polarization; any such inference needs additional assumptions.

• Learned. Zhang and colleagues (2018) Zhang et al. 2018 train a CNN on synthesized $T+R$ composites to predict the transmitted layer directly, using a perceptual loss so that the recovered $T$ looks like a real photograph rather than merely fitting pixels. This is L7.6show in its native habitat: the hand-built priors above (defocus, sparsity, ghosting) replaced by a prior fit to data.

The plain point: "$I = T + R_{\text{refl}}$ has many answers." Every method here is just a different way of inserting an assumption that the reflection is somehow distinguishable: blurrier, doubled, edge-sparser, polarized, or simply unlike a natural transmitted scene. It is a cousin of intrinsic images: instead of factoring light from surface, we are un-summing one illumination from another.

24.1.4 Shadow detection and shadow removal🔗⧉

What is a shadow, intrinsically? It is a region where the shading $S$ drops (less light reaches the surface) while the reflectance $R$ is unchanged. Same paint, just darker. That single observation places shadow removal squarely under intrinsic images: it is an edit on the shading layer only: flatten the darkening, keep the texture and markings untouched. If you could perfectly factor $I$ into $R\cdot S$, shadow removal would be trivial; the difficulty is the usual one, telling the two layers apart at the shadow's edge.

So the crux is detection: distinguish a shadow edge (a shading discontinuity, same surface, less light) from a material edge (a reflectance discontinuity, the surface itself changes). Several cues separate them:

• A shadow edge changes brightness but not hue much: it is the same surface on both sides, so its color stays put while its lightness drops. A material edge changes color outright.
• A shadow edge is often soft (a penumbra, where the light source is gradually occluded) and geometrically smooth, tracking the shape of the occluder rather than the texture of the surface.
• Shadowed regions are lit by skylight (bluer) rather than by direct sun (warmer), so a shadow often carries a faint color tell: the shaded side is not just darker but cooler.

Once the shadow boundary is found, removal is a gradient-domain edit of exactly the kind in Chapter 10.8, Poisson image editing L10.4show. In log intensity, label the gradients that lie on the shadow boundary (the big step from lit to shaded) and zero them out, leaving every other gradient (the surface texture, which is small and lives inside both regions) intact. Then Poisson-reconstruct the corrected log image with an intensity anchor and exponentiate. Ideally this preserves texture ratios while removing the illumination step; missed penumbra gradients, texture crossing the boundary, and noise prevent an exact separation. This is the same machinery as gradient-domain compositing, and indeed the outline of Chapter 10.8, Poisson image editing, flagged shadow removal as one of the "change only the guidance field" edits.

Two subtleties keep this grounded. First, the penumbra. A hard binary shadow mask, zeroed at a crisp boundary, leaves a dark halo where the real shadow softly faded, so you need a soft shadow matte, and the correction must follow the real penumbra. The fast bilateral solver (Barron and Poole, 2016) is one edge-aware tool to refine the matte: it snaps the soft correction to the image's own edges, which is precisely the edge-aware affinity idea L10.1show, pixels that look alike should get a similar correction. Second, the correction is multiplicative, not additive. The shadowed region received less light by some factor, so you recover it by scaling the shadowed radiance back up by the recovered shading ratio, a multiply in linear light L2.6show, not a brightness offset. Add a constant instead of multiplying and the lifted shadow will not match the surrounding tone.

The easy multi-image version is, once more, flash/no-flash (the added flash supplies another lighting condition, though it may cast its own shadows and need not erase the ambient ones), in Part 35, Computational illumination; and the learned single-image shadow removers are in Chapter 7.3, Deep learning.

24.1.5 Specular-highlight removal / "fake polarization" (the dichromatic model)🔗⧉

Glossy surfaces (skin, leaves, plastic, wet asphalt) show specular highlights: bright spots that are the light source reflected off the surface, not the surface itself. For many dielectrics under a single illuminant, the interface reflection approximately carries the illuminant color; metals, coatings, and mixed lights can violate this assumption, and it blows out texture, hides detail, and wrecks any downstream estimate of color or material. We would like the matte image: the diffuse-only version, as if the gloss had been wiped off.

The key is the dichromatic reflection model (Shafer 1985) Shafer 1985, which says that the light leaving a glossy surface is the sum of two terms with two different colors: a diffuse (body) term, light that entered the material and scattered back out carrying the surface's color; and a specular (interface) term, light that bounced straight off the surface without entering, carrying the illuminant's color. Per pixel,

where $\Lambda_{\text{body}}$ is the (fixed) surface color direction, $\Lambda_{\text{illum}}$ is the (fixed) illuminant color direction, and the scalar weights $m_d(x), m_s(x)$ (how much diffuse and how much specular) vary from pixel to pixel with the geometry. The point: across a single-colored glossy patch the weights change everywhere, but there are only two color directions in play.

That is what makes the separation possible, and the geometric picture is the clearest way to see it (Figure 24.1.5⛶ fullscreen). Plot the pixels of a glossy, uniformly-colored patch as points in RGB space. The dichromatic model confines them to the span of two color directions. Many patches populate an "L" or dog-leg within that plane: a line running along the body color $\Lambda_{\text{body}}$ (the matte shading, where $m_s = 0$ and only $m_d$ varies) with a spur branching off toward the illuminant white $\Lambda_{\text{illum}}$ (the highlight, where $m_s$ climbs). Lee (1986) Lee 1986 and, with the full cluster analysis, Klinker, Shafer and Kanade (1988) Klinker et al. 1988 showed how to identify the two arms of the dog-leg and then project each pixel back onto the diffuse line (subtract off the specular component along $\Lambda_{\text{illum}}$) to recover the matte image.

This is why the technique is called "fake polarization." The real way to kill a specular highlight is a polarizing filter on the lens: dielectric specular reflection can be strongly polarized at suitable angles, so a rotated analyzer can attenuate it; the effect is not universal. Doing the same job purely computationally, from one image via the dichromatic model gives you the polarizer's matte look without the filter and without the extra exposure, hence "fake." (There is a bonus: because the highlight carries the illuminant's color, it doubles as a free white reference. Lee 1986 used exactly this for color constancy, a suitable unsaturated highlight can constrain light color under the model, which is the same unclipped-highlight cue discussed in multi-illuminant white balance above.)

And this is its place in the chapter's frame. The specular term is an additive contamination of the shading layer, in the wrong color: the illuminant's rather than the surface's. Strip it and what remains is the clean diffuse $R\cdot S$. The cue is different from everything before (a color-direction prior rather than a gradient or smoothness prior) but the goal is identical: un-mix a layer that the camera summed in. As with reflection removal, the separation must happen in linear light, where the dichromatic sum is physically additive L4.6show; the genuine optical version (a rotating polarizer) and multi-light specular separation are in Part 35, Computational illumination, and the learned separators are in Chapter 7.3, Deep learning.