35.3 Illumination-based matting⧉
The matting equation is the same one used in the compositing chapter: every observed pixel is a mixture of a foreground color $F$ and a background color $B$ in proportion $\alpha$,
and pulling a matte means inverting it, recovering $\alpha$, $F$, and $B$ from $C$ alone, which is seven unknowns from three knowns per pixel and therefore impossible without help. The natural-image matting of Compositing, segmentation and matting bought that help with priors and user input. The computational-illumination answer is different and, where you can arrange it, far cleaner: change the capture so that some of the unknowns are simply known. If you control the background's color, $B$ is known. If you control the background's spectrum, you can route it to its own film. If you control the timing or wavelength of the lighting, you can take a picture in which the foreground and background are trivially distinguishable. Each method adds measurements or constraints that make matting easier, the mirror image of natural matting: spend a controlled light or backdrop instead of a prior.
35.3.1 The well-posed case to beat: chroma key⧉
The technique to keep in mind throughout is the blue/green screen, developed at length in Compositing, segmentation and matting and recapped in one line here: shoot the subject against a known, uniform, saturated background color chosen to be absent from skin, hair, and wardrobe, so that $B$ is known everywhere and only $\alpha$ and $F$ remain, leaving four unknowns for three color measurements. A keyer still needs a foreground-color model or other constraints; knowing $B$ alone does not determine $\alpha$ (Smith and Blinn 1996). Green dominates today (Bayer devotes more samples to green, and green is often separable from skin tones; noise and wardrobe constraints depend on the particular capture). Its failure modes are precisely the ways "known $B$" breaks: spill (green light bouncing onto the subject's edges and hair), uneven screen lighting ($B$ not actually one constant color across the frame), and genuinely soft edges (a hair boundary whose $\alpha$ is fractional even against a perfect screen). The entire history below is a series of attempts to keep the benefit of a known background while removing one or another of these failure modes, most dramatically by removing the colored screen from the set altogether.
35.3.2 The magic prism: Disney's sodium-vapor process⧉
The most beautiful illumination-based matte predates digital photography entirely. In the early 1960s, Disney perfected a sodium-vapor (or "yellow-screen") traveling-matte process, refined by Petro Vlahos and Ub Iwerks (Sodium-vapor process (Mary Poppins) 1964), and used to composite live actors into animated worlds in Mary Poppins (1964); it won a technical Academy Award. Its central trick is a piece of optics simple enough to state directly.
Light the backdrop behind the actors not with white light but with low-pressure sodium lamps, the same fixtures that once lit highways orange. Their emission is concentrated in the narrow sodium doublet near 589 nm. Light the actors with ordinary white light that contains everything except that sodium line. Now the scene carries two spectrally separated signals: the actors reflect a broad spectrum, while the backdrop glows in one pure wavelength. The camera design does the separation. A single lens forms one image, but behind it a prism / dichroic beamsplitter splits the light into two beams going to two films (Figure 35.3.1). One beam goes to ordinary color "taking" film carrying the actors' full-color image, with the sodium line filtered out so the backdrop reads as black; the other beam passes a narrowband filter tuned to the sodium line and lands on a black-and-white matte film that sees only the glowing backdrop, bright where the backdrop is visible and dark where an actor occludes it. That second film is the matte: a frame-by-frame silhouette of the actors, captured simultaneously and in perfect registration with the color image because both come from the same lens and the same instant.
Why was this better than a blue screen of its era? Because the matte is pulled by wavelength, not by color difference, it sidesteps two common blue-screen problems of the day: colored spill and the loss of blue or cyan in the subjects' own wardrobe. A monochromatic backdrop is far easier to make uniform (one lamp type, one wavelength), and a narrowband filter rejects it cleanly. The cost is exotic hardware: special sodium lamps, a custom beam-splitting camera (famously, very few were ever built), and dual film stocks in precise registration. The idea is to encode the background in a channel the foreground cannot occupy, then physically split that channel off to its own sensor. It reappears, modernized, as infrared matting.
35.3.3 Near-infrared and time-multiplexed matting⧉
The sodium process's deepest idea, a backdrop in a channel the eye and the color image ignore, becomes practical and flexible in the digital era through near-infrared (NIR) matting, used in Debevec et al.'s 2002 lighting-reproduction system Debevec et al. 2002 (lighting reproduction) and related systems. Replace the visible sodium backdrop with an infrared one: a backdrop made of retro-reflective material flooded by IR illuminators, or IR-emitting panels. The infrared illumination is invisible to the eye and rejected by the visible camera, but the physical backdrop can remain visible. The benefit is that it need not cast green or blue spill. A second, IR-sensitive camera (or an IR-filtered channel), co-located and registered with the color camera, sees the backdrop glowing uniformly bright: a built-in matte source with no saturated visible backing light to spill onto the subject, and none to relight the set (Figure, referenced). You get the constrained-$B$ benefit of chroma key, namely a known, uniform, easily separated background, without a green wall in the shot. Hair and translucent materials can have different opacity in IR and visible light, so an IR matte is not automatically the correct visible-light alpha.
A close relative is time-multiplexed lighting: instead of separating foreground and background by wavelength, separate them by time. Surround the subject with controllable lights and a controllable backdrop and interleave illumination conditions across successive frames at high speed to reduce, though not eliminate, inter-frame motion. One frame lights the subject for the beauty plate, an interleaved frame lights (or extinguishes) the backdrop for a clean matte frame or a clean plate, and the conditions are demultiplexed and combined in post. The same logic as the sodium camera, capture the foreground and the matte under separable conditions, in registration, is realized in the time axis with programmable lights rather than in the spectral axis with a prism. (This time-multiplexed machinery is the matting cousin of the light stage in Chapter 35.5, Light domes, which uses the same array of controllable lights to capture not a matte but a reflectance field.)
35.3.4 Flash/no-flash matting: separation by falloff⧉
The cheapest illumination-based matte needs no special backdrop at all, only the flash already on the camera and the inverse-square law. A flash brightens the near foreground enormously and the far background almost not at all, because irradiance from a point source falls off as $1/d^2$: a subject at one meter and a wall at five meters differ in flash illumination by a factor of twenty-five. So capture a flash and a no-flash frame, as in Chapter 35.1, Flash photography, and difference them. The difference image is large on the foreground (which the flash lit) and near zero on the background (which the flash barely reached), giving a foreground/background separation that depends on distance, not color (Sun et al. 2006) (Figure 35.3.2). The difference constrains foreground support but is not itself alpha: it also depends on reflectance, shading, and flash strength. Sun et al. use joint Bayesian flash matting to estimate soft edges from the pair; related refinements are discussed in Compositing, segmentation and matting, and you have a matte pulled from two ordinary frames of a subject standing in front of an ordinary, uncontrolled background. This is the technique Compositing, segmentation and matting, previews under "easy matting II"; here it belongs with the controlled-illumination methods. Multi-flash variants (the rig from Flash photography#Ramesh's multiflash) add the occlusion shadows cast at depth edges as a second, geometric cue to where the foreground boundary lies.
35.3.5 Engineering the capture instead of the prior⧉
Across the four techniques, the shared principle is clear. Green screen controls the background's color; the sodium prism controls its spectrum and physically splits it to a second sensor; NIR / time-multiplexed matting controls a channel or time slot the visible image ignores; flash matting controls the illumination falloff so distance becomes the separator. In each, an unknown that natural matting had to infer from a prior, chiefly $B$ or the foreground/background assignment, is instead made known by engineering the capture. This is the exact complement of the natural-matting program of Compositing, segmentation and matting, which spends user input and learned priors against an uncontrolled photograph; here we spend a controlled light, backdrop, or extra exposure, and get better-constrained measurements. Registration, transparency, spill, spectral mismatch, and foreground models still determine whether the matte is recoverable. Which road you take is a question of access: a film stage or a studio can build the perfect backdrop; a phone in someone's hand must lean on the prior. The matting equation is the same; only the budget differs.