Lecture Notes 1 — Images, C++🔗⧉

On the desktop: Python, C++, and Halide🔗⧉

Python is a convenient default for numerical experiments. NumPy represents images as arrays, while PyTorch and JAX add GPU computation and automatic differentiation. A per-pixel loop in Python can be slow; an array expression delegates the loop to compiled code. Vectorization can also create large temporary arrays, so memory traffic still matters. In conventional CPython builds, the global interpreter lock limits simultaneous execution of Python bytecode; compiled libraries can release it, and optional free-threaded builds have different behavior. Check the interpreter and extensions you deploy (Python documentation).

C++ offers explicit control of data layout, allocation, and parallel execution. That is useful for a pipeline with a tight latency or memory budget, but it requires more care with indexing, ownership, and build configuration. Standard containers and resource-owning objects reduce the need for manual allocation. Static types catch some errors; they do not catch a wrong color space or an out-of-bounds access automatically.

Halide separates an image-processing algorithm from its schedule: where and in what order values are computed, stored, vectorized, and parallelized. This lets the same algorithm support several execution strategies without rewriting its mathematical definition. Good schedules can avoid intermediate images and repeated memory transfers. They do not guarantee a fixed speedup over well-optimized C++ (Halide).

Much of the computational-photography literature was prototyped in MATLAB. For years it was the platform for this kind of work: a strong numerical and linear-algebra library, and an interactive IDE that could read, display, and manipulate images natively: an image was just a matrix, imread/imshow and array slicing made experiments effortless, and the Image Processing Toolbox supplied the rest. NumPy carries the same array-as-image approach into Python, while frameworks such as PyTorch combine tensor operations with automatic differentiation and GPU execution. We mention it because older papers and code you will meet are written in it, and the array-thinking it taught carries over directly to NumPy. Choose the environment that lets you reproduce and inspect a method, rather than translating working research code only for consistency of language.

For example, NumPy's out = a + 0.5*b makes a point operation almost as direct to write as its mathematical definition. That short edit-and-run cycle matters when the algorithm itself is uncertain. Once the operation is settled, C++ can give more control over storage and execution, with a heavier build cycle and more responsibility for object lifetimes. Halide adds a scheduling language when the performance question is where to compute and retain intermediate values. These are different costs to pay at different stages of a project.

Problem sets🔗⧉

Implementing a method exposes choices that an explanation can hide: boundary handling, units, numerical precision, and failure cases. These assignments from MIT's Digital & Computational Photography course ask you to build simplified algorithms and try them on real photographs. Their sequence follows the course rather than the book; each assignment links to the chapters it uses. Full handouts are collected in Part 45, Appendices.

Start with small inputs you can check by hand, then inspect the photographic results. A constant image, an impulse, or a known geometric transform often reveals an error more clearly than a natural photograph. If you are taking a course, follow its own submission and AI-assistance rules.

• 
  PS0: Environment and C++ basics. Set up the toolchain, write a minimal Image class, and read and write image files. (→ Chapter 9.1, Developing, Testing and Debugging)
• 
  PS1: Image class, point operations, color. Pixel access; brightness and contrast; the luminance–chrominance (YUV) split; gamma; white balance; and the Spanish-Castle afterimage illusion. (→ Chapter 8.1, Image representation, Chapter 8.3, Tone point operations, Chapter 3.2, Perceptual color and trichromatic vision)
• 
  PS2: Convolution and the bilateral filter. Box and Gaussian blur, image gradients, sharpening, and bilateral denoising (with a YUV chroma variant). (→ Chapter 6.2, Convolution, Chapter 10.1, Bilateral filtering)
• 
  PS3: Denoising and demosaicking. Align-and-average a burst (and the ISO/variance story); Bayer demosaicking: basic green, then edge-directed green and color-difference R/B; and Prokudin-Gorsky three-plate color alignment. (→ Part 13, Denoising, Part 15, Demosaicking and joint methods)
• 
  PS4: High dynamic range. Merge an exposure bracket into a radiance map and tone-map it for display. (→ HDR merging, Chapter 18.2, Global tone mapping)
• 
  PS5: Resampling, warping, and morphing. Nearest / bilinear / bicubic / Lanczos resampling; Beier–Neely segment warps; and a full Beier–Neely morph. (→ Chapter 11.1, Warping, Chapter 11.3, Morphing)
• 
  PS6: Homographies and manual panoramas. Homogeneous coordinates, warping by a homography, solving the homography from four point correspondences, and stitching a planar mosaic. (→ Chapter 20.1, Manual panorama stitching from multiple views)
• 
  PS7: Automatic panoramas. Harris corners (the structure tensor), patch descriptors, the second-nearest-neighbor ratio test, RANSAC, and feathered / two-scale blending. (→ Chapter 20.2, Automatic panorama stitching from multiple views and feature matching)
• 
  PS8: Non-photorealistic rendering. Paintbrush splatting and single-scale, two-scale, and gradient-oriented painterly rendering. (→ Chapter 25.2, Non-photorealistic rendering)
• 
  PS9: Make-your-own, video, and ethics. A self-proposed project (optical-flow retiming, phase-based video magnification, and the like) together with the ethics deliverable. Its open menu of advanced topics seeds many sections of Part 33, Computational optics and coded imaging. (→ Chapter 12.8, Optical flow, Chapter 38.3, Motion and video magnification)

The assignments deliberately distill an area into a small algorithm that you can build and inspect. Their purpose is to expose the principle before the production machinery obscures it. A basic align-and-average burst, for example, makes registration and noise reduction visible even though a shipping camera will need more elaborate handling of motion and outliers.

The pixel integral🔗⧉

A pixel does not sample a single ray; it integrates over four groups of variables, each weighted by how the pixel responds there (Figure 2.4.2⛶ fullscreen):

1. the aperture: all the ray directions $(\theta, \phi)$ the lens admits toward this pixel, a solid angle set by the f-number;
2. the pixel's area $(x, y)$ on the sensor, a finite patch, not a mathematical point;
3. wavelength $\lambda$, weighted by the sensor's and color filter's spectral response; and
4. the exposure time $t$, the interval the shutter is open.

The spectral axis, $\lambda$, is the one this book develops furthest elsewhere: collapsing a whole spectrum onto a few numbers weighted by $r(\lambda)$ is exactly where color is born, and how a sensor is built to make that measurement in three or more channels, together with what the resulting numbers mean perceptually, is the subject of Part 3, Visual perception and color and Part 4, Color spaces and color technology, and in particular the chapter on Chapter 4.6, Sensing color: multiplexing strategies. Here we treat $\lambda$ simply as one more axis the pixel integrates over.

Schematically, the value recorded at a pixel is

the scene radiance integrated over aperture × pixel area × wavelength × time, weighted by the pixel's spectral response $r(\lambda)$ and the geometric weight $k$ that the aperture and footprint impose. Measurement is integrating the plenoptic function against the pixel's sampling kernel, and the support of that kernel along each axis is exactly a photographic knob. Because the underlying capture counts photons, this integral is linear in scene radiance, a fact we will need both for high-dynamic-range merging and for the noise analysis in Chapter 2.8, Noise, signal-to-noise ratio and dynamic range.

The four domains are the four knobs. This matters because each integration domain is governed by a control the photographer actually sets: the integral is the camera's exposure settings, written as mathematics:

• the aperture integral is set by the f-number $N = f/D$: stopping down shrinks the cone of directions each pixel sums over (less light, more depth of field); opening up widens it;
• the exposure-time integral is set by the shutter speed: a longer interval integrates over more time (more light, but more motion blur);
• the pixel-area integral is fixed by the sensor's pixel pitch (the physical size of a photosite) so larger photosites collect more photons at equal sensor irradiance and exposure, while larger sensor area does not by itself imply larger pixels;
• the wavelength integral is shaped by the spectral response (the sensor's sensitivity curve times the color filter over each photosite) which collapses a whole spectrum into a single number per channel.

The one familiar exposure control missing from this list is ISO, and the omission is instructive: ISO is not an integration limit. It scales the result of the integral after the light is collected (analog or digital gain), rather than changing how much light is gathered, which is exactly why pushing ISO amplifies noise along with signal, as the noise section will show.

The deeper reason to write capture this way is that each of the four integration domains earns its own topic later, and much of photography is the craft of choosing how to shape these four integrals. Every later sensing topic is just what happens when you change the support of one axis:

• integrating over the aperture is what produces depth of field (the lens chapter): a wide aperture integrates over a fat cone of directions, so an out-of-focus point spreads its energy across many pixels: the circle of confusion. The light-versus-blur trade lives entirely on this one axis;
• integrating over the pixel's area is sampling and antialiasing (Basic Image Processing): a finite footprint makes each sample a box-filtered average rather than a point sample, pre-blurring the image before it is laid on the grid: a wider footprint suppresses aliasing at the cost of resolution;
• integrating over time is motion blur (Motion/Video): a long exposure sums a moving scene over the interval, the temporal sibling of the aperture, trading light against sharpness;
• integrating over wavelength is color sensing: the spectral response collapses a whole spectrum into one number per channel, exactly as a cone does (Chapter 3.2, Perceptual color and trichromatic vision). Because a color-filter array gives each photosite a different $r(\lambda)$, color is spatially multiplexed across the Bayer mosaic and must be reconstructed by demosaicking.

The rest of the sensing story reduces to one question, which kernel, how wide, and what does it lose: antialiasing widens the spatial kernel, depth of field is the aperture kernel, motion blur is the time kernel, and demosaicking inverts the wavelength multiplexing. Each is also a source of noise: a narrower kernel along any axis (a smaller footprint, a shorter exposure, a stopped-down aperture) integrates fewer photons, and fewer photons mean a noisier estimate, which is exactly where Chapter 2.8, Noise, signal-to-noise ratio and dynamic range picks up.

All four axes are then summed into one number: the conventional sensor couples them, discarding how the light was distributed across directions, area, color and time alike. The ideal sensor would do the opposite: decouple the axes, measuring each one separately so we could decide after the shot how to combine them. Chasing that decoupling is a whole branch of computational photography, which the end of the chapter frames.

Framed this way, the measurement integral is a general language for imaging. Every imaging system, from a pinhole to a light-field camera, is described by which weighting it applies to the plenoptic function before integrating: the choice of weights over direction, area, wavelength, and time is the design of the system. And because integration is linear, every such system is a linear operator on the plenoptic function, the viewpoint that Chapter 6.7, Image formation as a linear system develops in full.

For the sake of simplicity, we routinely speak of imaging in terms of infinitesimal quantities: a pinhole that admits a single ray, an exposure that freezes an instant, a pixel that samples a point, a single wavelength of light. These idealizations keep the geometry clean and the diagrams legible. But no real measurement is infinitesimal. A sensor reports a number only by integrating over a finite aperture, a finite area, a band of wavelengths, and a stretch of time, because every axis of the plenoptic function must be summed over to gather enough photons to register at all. The idealized ray and the frozen instant are the limit the mathematics reaches for; the finite integral is what the camera actually delivers, and the gap between the two is where depth of field, motion blur, color mixing, and noise all live.

Off-axis ($\cos^4$) falloff. One prediction falls out of this irradiance integral. Rays reaching the corners of the frame arrive obliquely, so the same scene radiance delivers less irradiance there: the aperture-and-area geometry of the pixel integral works out to a $\cos^4\theta$ dimming toward the edges, called natural vignetting (Figure 2.4.3⛶ fullscreen). It is distinct from optical vignetting (lens elements physically blocking edge rays, covered in Optics) in that natural vignetting is a clean consequence of the pixel-integral geometry alone.

To interpret the integral as expected photoelectrons, let $r(\lambda)$ include quantum efficiency and the conversion from radiant energy to photons, $\lambda/(hc)$. If $r$ denotes quantum efficiency alone, that factor must appear explicitly. Optical transmission and the projected-area factor belong in $k$. Fixing these weights makes the expected measurement linear; the observed count is random, and saturation or later encoding breaks this simple linear relation.

Exposure values (EV) and stops🔗⧉

The pixel integral hands back a linear amount of light: double the aperture area or double the time and you double the number. Photographers almost never speak that way. They count light in stops, where one stop is a factor of two, because both the eye and the medium respond to ratios rather than differences (the same logarithmic story behind gamma encoding and Weber's law, Chapter 3.2, Perceptual color and trichromatic vision). A stop is the natural quantum of exposure: one doubling, one halving, one bit of light. Written on the log$_2$ scale the integral lives on, the whole messy business of exposure becomes simple addition and subtraction.

The power of the unit is that it is the same unit on every axis of the integral. One stop of aperture (a $\sqrt2$ step in the f-number, since light goes as $1/N^2$, Chapter 2.3, Lens image formation), one stop of shutter (a factor of two in time $t$), and one stop of ISO (a factor of two in gain) all change the brightness by the identical factor of two, so they can compensate one another in output brightness over an unclipped, approximately linear range. They do not preserve motion blur, depth of field, photon count or read-noise behavior. That commensurability is exactly what lets the three trade against one another, the exposure triangle: give back a stop of light on one leg, take it up on another, and the intended output brightness can remain unchanged. Many different aperture-shutter-ISO combinations therefore land on the same brightness, a redundancy photographers call equivalent exposures and rely on constantly (the faucet-and-bucket reciprocity of Chapter 5.2, Exposure and capture timing).

To name a whole family of equivalent settings with a single number, photography uses the exposure value (EV):

the base-2 logarithm of squared f-number divided by exposure duration expressed in seconds. EV describes aperture and shutter settings; relating it to scene luminance also requires ISO and meter calibration. One EV is one stop. Every $(N, t)$ pair on the same EV describes the same exposure, so a single EV names an entire diagonal of the aperture-shutter grid:

Read across: each stop you close the aperture is paid back by one stop of slower shutter, and the exposure never moves. Because EV folds the settings into one number, it doubles as a measure of scene light. At ISO 100 the same EV names how bright the scene is (sometimes called the light value): a meter's reading, $L = K N^2/(t S)$, is EV in disguise, and the old sunny 16 rule ("bright sun is $f/16$ at 1/ISO") is nothing but a printed EV chart, one EV per lighting condition.

Once you have the unit, it turns up everywhere, which is the real reason to learn it. Meters report in EV, and every metered scene is placed on the EV ladder relative to mid-gray. Exposure compensation is dialed in EV ($+1$ EV to keep a snowfield white, $-1$ for a black cat), a direct nudge along the ladder. Auto-bracketing shoots frames at, say, $-2, 0, +2$ EV to merge into one high-dynamic-range image (HDR merging). A scene's dynamic range, a sensor's, a display's, are all quoted in stops (the ratio of brightest to darkest, in powers of two). Neutral-density filters are sold in stops (an ND8 is three stops), a lens's speed is "one stop faster," flash and studio lights meter in stops, and the noise story is a stop story too: because shot noise is set by photon count, every stop less light collected costs half the photons and a predictable step of signal-to-noise ratio (Chapter 2.8, Noise, signal-to-noise ratio and dynamic range). Stops and EV are the shared logarithmic language in which exposure, metering, dynamic range, filtration, and noise are all spoken; the full exposure and metering practice built on them is Chapter 5.2, Exposure and capture timing.

Photon to number: the photosite🔗⧉

A digital sensor is a two-dimensional array of photosites, one per pixel. Each photosite is a small potential well that counts photons: arriving light frees electrons, the well accumulates that charge, and at the end of the exposure the charge is converted to a voltage and then, by an analog-to-digital converter (ADC), to a number. Because the well literally counts arrivals, the number it reports is, to good approximation, linear in the amount of light that fell on it. That linearity is not a minor technical detail: it is what lets us treat a raw image as a calibrated radiometric measurement, and it is the foundation that makes HDR merging, deblurring, and depth estimation meaningful later in the book.

Two revolutions are often conflated. The first was electronic imaging: turning light into an analog electrical signal (the era of the CCD and analog video). The second, the one computational photography lives on, was digital: putting an ADC in the path so the signal becomes numbers we can compute on. A sensor that captures light electronically but never digitizes it is a television camera, not a computer's eye.

Above each photosite sits a microlens that funnels light onto the light-sensitive area, necessary because wiring occupies part of each pixel, so the fill factor is less than 100%. Under the microlens sits a color filter: a bare photosite is colorblind (it only counts photons), so a mosaic of color filters, most commonly the Bayer pattern, is laid over the grid and full color is reconstructed later by demosaicking. That is only the briefest mention here; how a colorblind sensor is made to see color, and the several ways to multiplex the measurement, is the subject of its own chapter in the next part, Chapter 4.6, Sensing color: multiplexing strategies (§4.2). Finally, the map from scene radiance to the rendered pixel value is the camera response curve: raw is roughly linear, while a JPEG has a non-linear curve (gamma plus a tone curve) baked in: recovering that curve is exactly the problem HDR has to solve.

How big is a photosite, really? Pixel sizes span more than a factor of ten across cameras, and the range maps directly onto the sensor-size story below. A flagship phone crams its pixels down to about $0.7$–$1.0\,\mu\text{m}$; a 1-inch compact sits near $2.4\,\mu\text{m}$; APS-C runs $3.5$–$4\,\mu\text{m}$; a full-frame sensor is $4$–$6\,\mu\text{m}$; and the largest-pixel cameras (low-resolution full-frame bodies, astronomy and scientific sensors) reach $8\,\mu\text{m}$ and beyond. The telling comparison is biological: a foveal cone is about $2\,\mu\text{m}$ across (Chapter 3.2, Perceptual color and trichromatic vision), so a big camera pixel is several cones wide, while a modern phone pixel has shrunk below the size of a single cone (Figure 2.7.4⛶ fullscreen). Size matters here because a larger collecting area receives more photons at equal irradiance and exposure, increasing shot-noise standard deviation but reducing noise relative to signal, which is the whole reason larger sensors look cleaner (→ The impact of sensor size, below). The phone's tiny wells force it to make up the deficit computationally, with burst capture and denoising rather than glass and silicon.

The eye and image formation🔗⧉

Look from a distant building to a book in your hand. The eye changes focus while directing its highest-resolution region toward the words. Its optics obey the same refraction laws as a camera, but its sampling and processing differ substantially (Part 2, Fundamentals).

Light entering the eye passes through the cornea and the lens, the eye's two refracting elements, the same Snell's-law optics as a camera lens (the cornea does most of the bending; the lens fine-tunes focus by changing shape, accommodation). The pupil, the iris's adjustable opening, is the eye's aperture. The image lands on the retina, a thin sheet of neural tissue lining the back of the eyeball (Figure 3.1.1⛶ fullscreen).

Rods and cones supply most of the signals used for image-forming vision. Rods support vision at low light levels; because they share one spectral sensitivity, rods alone cannot distinguish wavelength from intensity. Cones support daylight color vision through three receptor classes. At intermediate levels, rods and cones both contribute. Their distribution is nonuniform: cones are densest near the center of gaze, while the central foveola lacks rods (Figure 3.1.2⛶ fullscreen). Looking slightly away from a faint star can therefore make it easier to detect.

Foveal cones are only a few micrometers across, with spacing near $2.5\,\mu\text{m}$ in a dense central mosaic; sizes and spacing vary across observers and retinal position. Away from the fovea the cones grow to several micrometers and thin out, while rods, about $1$ to $2\,\mu\text{m}$ across, fill the spaces between them; in Curcio and colleagues' measurements, cone density peaks near 200,000 per square millimeter at the foveal center and rod density near 150,000 per square millimeter about twenty degrees out Curcio et al. 1990. Dense sampling supports fine detail, but does not alone determine acuity. Optical blur, neural pooling, contrast, and illumination also matter. A camera photosite and a cone of similar size need not have similar sensitivity or angular resolution.

The gap at the optic disc is worth more than a label, because it is the first and cleanest evidence that what you see is constructed. Each eye has a patch of its visual field about five degrees across, some fifteen degrees to the ear side of where you are looking, that delivers nothing at all, and you have never once noticed a hole there. Two things hide it. The other eye covers that part of the field, so closing one is enough to remove the first explanation. What remains is the second and the interesting one: the missing region is filled in with whatever surrounds it, so a line broken exactly there is seen as continuous and a textured field closes over the gap. Nothing in the retina reports the fill; it is manufactured, and it is indistinguishable from the real thing (Figure 3.1.4⛶ fullscreen).

S cones are sparse and absent from a small region at the very center of the fovea. Short-wavelength detail also suffers from the eye's chromatic aberration. Fine luminance detail is carried mainly by combined L- and M-cone signals, while purely chromatic variations have lower spatial resolution. This does not make a green channel identical to luminance, or imply that blue detail is universally sharp through filling-in. The relative importance of luminance detail helps motivate the extra green samples in the Bayer pattern (Chapter 4.6, Sensing color: multiplexing strategies).

The retina processes signals before sending them along the optic nerve. Photoreceptors connect through intermediate neurons to ganglion cells. Many ganglion cells compare a central region with its surround, making them sensitive to local contrast; others have different spatial, temporal, or chromatic selectivity. Signals then reach several brain regions, including the lateral geniculate nucleus and visual cortex. A camera-to-retina analogy is useful for the optics, but retinal output is already a collection of processed signals.

Focus errors provide another connection to camera optics. In a simplified eye, myopia places the image of a distant point in front of the retina; hyperopia places it behind the retina when accommodation is relaxed. Astigmatism gives different focusing power along different orientations. Presbyopia reduces the range over which accommodation can adjust focus with age. These are different optical problems, not one generic blur.

Lens power is expressed in diopters:

where $f$ is the signed focal length in meters. A converging lens with focal length $0.5$ m has power $+2$ diopters; a diverging lens of $-0.5$ m has power $-2$ diopters. This describes the lens, not a complete eye prescription: cylinder power and orientation also matter for astigmatism.

An autorefractor estimates refractive error from light returned by the eye; subjective comparison with trial lenses refines the result. A Shack–Hartmann sensor measures the slopes of the outgoing wavefront using a lenslet array, allowing more detailed aberration estimates. The shared imaging idea is to infer optical properties from the recorded light. Part 22, Compound lenses, and aberration correction develops wavefront measurement.

Light adaptation🔗⧉

Walk into a dark cinema and details gradually become visible. The visual system changes sensitivity as the prevailing light changes. A useful model of a saturating response is the normalized Naka–Rushton curve Naka & Rushton 1966,

At intensity $I=I_0$, the response is one half. Changing $I_0$ moves the operating range; $n>0$ controls the slope. This family can describe measured responses, but one curve and a moving average do not specify all of light adaptation. The curve is sigmoid when plotted against log intensity; for $n=1$ on linear axes it is a saturating hyperbola.

Figure 3.3.1⛶ fullscreen lets you change the operating point and compare axis scales.

Neural gain control, photopigment bleaching and regeneration, and pupil changes contribute on different timescales. A pupil diameter change from 2 to 7 mm changes area by $(7/2)^2\approx12$, much less than the full range over which vision operates. Full dark adaptation can take tens of minutes. Local adaptation also contributes to afterimages. In the Spanish-Castle demonstration, an adapting color image is followed by a grayscale version that briefly appears colored (Figure 3.3.3⛶ fullscreen); the effect is not an illusion of motion.

Vision is often described in three regimes. Photopic vision is cone-dominated and supports detailed color vision. Scotopic vision is rod-dominated and lacks ordinary trichromatic color discrimination. Mesopic vision combines rod and cone signals. The transitions depend on adaptation, spectrum, retinal position, and stimulus, so they are not universal luminance cutoffs.

As illumination falls, relative spectral sensitivity shifts toward shorter wavelengths, the Purkinje shift Purkyně 1825 (Purkinje shift). A red flower can become relatively darker than nearby blue-green foliage. This is a change in relative sensitivity, not a statement that all dim scenes look blue. Luminance (cd/m²) describes light from a surface in a viewing direction; illuminance (lux) describes light arriving at a surface. A scene's lux value alone does not determine the observer's adaptation state.

Retinal illuminance is often summarized in trolands: luminance in cd/m² multiplied by pupil area in mm²,

For example, a 100 cd/m² patch viewed through a 3 mm diameter pupil gives about $100\pi(1.5)^2\approx707$ trolands. This is a photometric convention, not a universal conversion to photons. For order of magnitude only, Rodieck's estimates put a rod under starlight at roughly one photoisomerization every several seconds, and a cone in a sunlit scene at around a million per second Rodieck, The First Steps in Seeing, a span the pupil, bleaching and neural gain together have to accommodate. Predicting photoisomerizations additionally requires the spectrum, ocular transmission, receptor sensitivity and collecting area, and integration time. A photon absorbed by a visual pigment can isomerize the molecule and initiate a graded receptor response; it does not necessarily cause a perceptible event.

At very low light levels, photon statistics and neural noise limit detection. Hecht, Shlaer, and Pirenne inferred that a detectable flash could produce only a few photoisomerizations distributed across many rods Hecht, Shlaer & Pirenne 1942. Recordings subsequently established that individual rods can respond to single photons Rieke & Baylor 1998. That cellular response must be distinguished from conscious detection. Tinsley and colleagues reported slightly above-chance detection of single photons incident on the cornea Tinsley et al. 2016 (humans detect a single photon), but later retinal and psychophysical work challenged the interpretation of that result (2024 study). These findings do not make the eye a reliable photon counter. Chapter 2.8, Noise, signal-to-noise ratio and dynamic range explains the corresponding photon-counting limits in cameras.

The range over which we can function after adapting is much greater than the range we can distinguish at one operating point. A bright scene and a dark cinema do not both fit into one unchanged receptor response. Adaptation moves that response so its useful slope falls near the current stimulus levels. This is why a claim about the eye's total dynamic range does not specify how many stops a display must reproduce simultaneously. HDR capture and tone mapping must account for the scene, the display, and the observer's adaptation separately.

Contrast🔗⧉

If vision reads ratios, its natural unit is contrast, and the oldest quantitative statement of that is Weber's law (Weber 1834 (Weber's law)): the just-noticeable increment in intensity is a roughly constant fraction of the background, $\Delta I / I \approx \text{const}$. The smallest brightness change you can detect grows in proportion to the brightness already there. A candle added to one candle is obvious; the same candle added to a hundred is nothing.

Weber's law is a statement about threshold. It describes the smallest increment you can just detect, and there it holds well across the middle of the photopic range, but it says nothing about how strong a large, plainly visible contrast looks. That suprathreshold regime has its own accounts, and two classical ones matter here. Fechner integrated Weber's law, treating each just-noticeable difference as one equal step of sensation, and reached a logarithmic relation between intensity and perceived magnitude, $\psi \approx k \log \phi$ Fechner 1860 (Elemente der Psychophysik). Stevens, from direct magnitude estimation, argued instead for a power law, $\psi = k\,\phi^{a}$, with an exponent that depends on the modality Stevens 1957 (power law). Which is right has been argued for a century and does not need settling here.

These models preserve different relations. A common scale leaves Weber contrast unchanged. Under Fechner's model it adds a constant to perceived magnitude, preserving differences; under Stevens's model it scales magnitude, preserving ratios. Neither implies unchanged brightness. Gamma encoding is a power law and does not turn a multiplier into an additive offset; a logarithm does.

For a small target on a uniform positive background $L_b$, use Weber contrast. For a periodic grating, use Michelson contrast:

The first measures a target's increment or decrement relative to its background. The second measures modulation relative to the midpoint of a periodic pattern's extrema. Both cancel a common positive multiplier, but they describe different stimuli. For luminances 80 and 120, Michelson contrast is 0.2; a 120 target on an 80 background has Weber contrast 0.5. Use linear luminance, not encoded pixel values.

Contrast is set by context too, not by a patch in isolation. A gray patch looks lighter on a dark surround and darker on a light one, the effect called simultaneous contrast (Chevreul 1839 (simultaneous contrast)), while the Adelson checker-shadow illusion adds cues about scene geometry and illumination. Meet it in its barest form before any tone-mapping operator arrives, because every one of them is a machine for changing exactly this (Figure 3.3.7⛶ fullscreen).

That much is compatible with a simple story: the response to a patch is divided by the light around it, so more surround means a darker-looking patch. Simple surround averaging cannot account for White’s illusion (Figure 3.3.8⛶ fullscreen), which is a counterexample to simple surround division: the patch that looks lighter is the one with more white against it, which is the opposite of what dividing by the surround predicts. Perceptual grouping and junction structure contribute beyond immediate border contrast.

Chromaticity diagram🔗⧉

For $X+Y+Z>0$, define

These coordinates discard scale: doubling the light leaves $x,y$ unchanged. Black has undefined chromaticity. Given $xyY$ with $y>0$, recover $X=xY/y$ and $Z=(1-x-y)Y/y$. Thus chromaticity plus luminance specifies the original tristimulus values; chromaticity alone does not.

Monochromatic lights trace the spectral locus. Nonnegative mixtures lie in its convex hull, closed by the line of purples. The reference white is a selected point within this region, not an intrinsic center of the diagram. Euclidean distance in $xy$ is not a perceptual color difference.

Geometrically, nonnegative spectra project to a cone of realizable tristimulus values Koenderink, Color for the Sciences. Normalizing by $X+Y+Z$ intersects each nonzero ray with a plane. Normalizing by another linear quantity gives a different cross-section. Although the underlying three-dimensional coordinates are related linearly, the normalized two-dimensional coordinates are related projectively.

MacLeod–Boynton chromaticity MacLeod & Boynton, cone-excitation chromaticity diagram (1979) uses cone fundamentals scaled so their L and M components sum to the luminance function. It plots $l=L/(L+M)$ and $s=S/(L+M)$. To interpret a horizontal displacement as an L–M trade at fixed luminance, hold the omitted scale $L+M$ fixed. The coordinates themselves do not force all plotted stimuli to have equal luminance, just as $xy$ does not.

A positive XYZ or LMS triple need not be physically realizable: positivity of three projections is weaker than existence of a nonnegative spectrum producing them. Reflectance introduces further constraints, since it must lie between zero and one under a specified illuminant. Do not confuse this surface-color problem with unconstrained emission.

Why divide by the sum rather than by luminance alone? Dividing by $Y$ is an appealing choice when the goal is to separate luminance from color, but coordinates such as $X/Y$ can become very large when $Y$ is small compared with another component. By dividing nonnegative XYZ coordinates by their sum, we instead put each normalized component between zero and one and make the three sum to one. The resulting diagram is bounded. This is a practical benefit of the chosen normalization, not a statement that its distances are perceptually uniform. Figure 4.1.8⛶ fullscreen makes the choice visible as a change of slicing plane.

The curved locus also has a physical origin. Sweep a monochromatic test through wavelength. At each setting its position comes from the three matching-function values at that wavelength, normalized by their sum. Those values vary together along the sensitivity curves, so the plotted point follows a curve rather than the sides of a triangle. Mixtures fill the convex region determined by that locus. The diagram's outline reflects the observer's spectral sensitivities; it is not an arbitrary shape chosen for a plotting convention.

The limits of white balance, and CRI (color rendering index)🔗⧉

Two illuminants can have the same white chromaticity yet render colored objects differently. If a source delivers little or no energy in bands that distinguish two reflectances, three channel gains cannot recover the missing distinction. Even smooth illuminants need not be related by an exact diagonal correction.

The color rendering index (CRI) compares specified test colors with a reference source at a similar correlated color temperature. The commonly quoted general index $R_a$ averages eight test colors; it is not a complete assessment of saturated colors, preference, or camera fidelity. A high score does not specify the illuminant spectrum. Contemporary lighting specifications may also report more detailed fidelity and gamut measures. For photographic work, examine the actual source and the materials that matter, especially when mixing lights.

A matrix fitted on chart patches can reduce their residuals without accurately correcting other materials. Separate fitting and evaluation sets when testing a camera transform. Neither a low training error nor a neutral gray row establishes general spectral recovery.

RGB color spaces🔗⧉

An RGB space chooses primary chromaticities and a reference white; its stored representation also specifies a transfer function. Different choices serve different devices and workflows. sRGB is a common SDR delivery space. Display P3 extends its red and green coverage. Adobe RGB extends especially into greens and cyans useful in some print workflows. Rec. 2020 has still wider primary chromaticities. ProPhoto is a large working space with imaginary primaries.

With fixed integer precision, a larger gamut generally spreads codes more widely in color. The effect varies across the gamut and depends on the transfer function. Use sufficient precision for editing, then choose the delivery space from the output requirements. A screen cannot display every working-space value merely because software can store it.

Display-oriented spaces describe a reference set of primaries. Working spaces can instead choose imaginary primaries to enclose more physical colors. ProPhoto’s green and blue are outside the spectral locus; its red is not the imaginary pair’s other member. Such coordinates are useful for calculation, but their nonnegative cube contains values no physical light can produce.

Camera raw is different. Its channels are spectral measurements, often one color sample per photosite in a mosaic, not weights of three emission primaries. A camera would admit an exact universal XYZ matrix if its sensitivities spanned the standard observer’s matching functions. Real cameras generally only approximate this condition, so color correction is fitted and depends on spectra and illumination. It is not meaningful to invert an arbitrary three-by-many sensitivity matrix and call the result three physical display primaries.

Camera profiles can provide multiple illuminant calibrations. Do not assume every DNG contains exactly two matrices at fixed temperatures; use the tags and conventions of the actual profile. Camera color-space menu settings usually govern rendered output and previews, while raw channel interpretation remains camera-specific.

The correction matrix also propagates noise. With equal independent input variances $\sigma^2$, output channel $i$ has variance $\sigma^2\sum_j M_{ij}^2$. More generally use $M\Sigma M^T$. Negative coefficients do not alone prove amplification; their magnitudes and the input covariance matter. The filter design and color fit balance sensitivity, color accuracy, and noise, as developed in Chapter 4.6, Sensing color: multiplexing strategies.

This distinction explains two different kinds of color-conversion experiment. Between linear RGB working spaces built from the same standard observer, convert a color to XYZ and then to the destination coordinates; the two matrices compose into one exact coordinate transform. For a camera, photograph a set of known reflectances under a specified light and fit a matrix from the measured camera responses to their target XYZ values. It can fit those samples well and still make errors on other spectra. Two spectra that the camera records identically cannot be separated by any later matrix, even if the standard observer distinguishes them. More training patches improve the fit over a chosen population; they do not restore spectral information absent from the three measurements.

The choice of a working gamut also affects quantization. With a fixed integer bit depth, a wider set of primaries spreads the available code combinations over a larger range of colors. It can retain a saturated color that a smaller space clips, yet give coarser steps in a region shared by both spaces. Increasing bit depth or using floating point reduces that pressure. This is why a space suited to intermediate editing need not be the best choice for compact delivery. The working space must retain the colors the pipeline needs; the output space must match the interpretation supported by the destination.

A caution about that diagram before we read too much into the triangle sizes: the $xy$ chromaticity plane is a space for measuring color, not for judging how different two colors look. It stretches the greens enormously and cramps the blues and purples, so neither the relative areas of these gamuts nor the distances between their primaries track perceived difference. Re-plot the identical primaries on a perceptually more uniform diagram, the CIE 1976 UCS ($u'v'$) space, and every triangle changes shape (Figure 4.4.3⛶ fullscreen): the relative areas and distances change. The newer diagram is more uniform, but it is still not a universal perceptual ruler. That $XYZ$ (and its $xy$ projection) is not perceptually uniform is precisely what will force the deliberately non-linear spaces of Chapter 4.5, Advanced color: opponents, perceptually-uniform, and preferred.

Two design processes can lead to an RGB space. In a display-oriented design, begin with the primaries a reference display can emit, then describe colors as amounts of those lights. In a working-space design, begin with the colors that editing should retain, then choose a coordinate system that encloses them. The latter need not correspond to a manufacturable display. An imaginary primary is acceptable as a coordinate axis because no lamp is being asked to emit it while the image is stored or edited. Physical realizability becomes an output constraint when those values are converted for a screen or printer.

This distinction explains why choosing a larger working space does not upgrade the monitor. It changes which intermediate colors can be represented without clipping to the working cube. Displaying them still requires conversion and, where needed, gamut mapping. Conversely, a delivery space can be chosen for compatibility with the intended viewing path even when a larger space was useful during editing. The export dialog asks for an engineering choice about that destination, not a ranking in which the largest triangle is always best.

The noise formula can be read numerically. A correction row with coefficients 1.8, -0.9, and 0.1 has squared length 4.06. With equal independent input noise, its output standard deviation is therefore about 2.01 times the standard deviation of one input channel. The row may improve the color fit while increasing that channel's noise. This is why a camera characterization should be evaluated for both reconstruction error and noise propagation, using the actual covariance rather than assuming that every color matrix has the same cost.

Preferred colors: skin, sky, and other memory colors🔗⧉

Viewers bring expectations to familiar objects, especially faces. A camera can produce a colorimetrically plausible image yet render a face in a way its subject dislikes. Skin rendering therefore requires both accurate measurement and a deliberate rendering objective.

The vectorscope’s conventional skin-tone line is a useful reference under some lighting and rendering conditions. It is not a universal target. Complexions, blood perfusion, body site, illumination, makeup, exposure, and camera processing all affect the measured hue. Pulling every face toward one line can remove real variation.

CIELAB skin measurements must specify their illuminant and observer. The individual typology angle Chardon et al. 1991 (ITA°) is a descriptor based on $L^*$ and $b^*$, conventionally expressed as $\mathrm{ITA}=\arctan((L^*-50)/b^*)$ in degrees in its intended domain. It discards $a^*$ and is not a complete color description. Do not infer a person’s complexion category from an uncalibrated JPEG or treat fixed hue limits as an inclusive skin detector.

Melanin and hemoglobin strongly influence skin reflectance, but skin is not exactly a two-parameter color surface. Hemoglobin oxygenation, layer thickness, scattering, and illumination also matter. Tsumura et al. Tsumura et al. 2003 estimate pigment-related image components using a model and statistical assumptions; arbitrary RGB attenuation vectors are not that calibrated method.

A physical forward calculation keeps the spectral integration explicit:

reflectance = skin_model(wavelengths, pigment_parameters, scattering_parameters)
stimulus = illuminant_spectrum * reflectance
XYZ = integrate(stimulus * XYZ_matching_functions, wavelength)
RGB = render_for_display(XYZ, viewing_conditions)

A log-RGB approximation can simplify separation under suitable conditions, but the logarithm of a spectral integral is not the integral of a logarithm. Label any reduced model and validate it against measured spectra. For spatially resolved rendering, subsurface scattering also matters: a BSSRDF relates entering light at one surface point to exiting light at another. Chapter 35.5, Light domes develops computational capture for face relighting.

Historical calibration choices also mattered. Shirley reference cards and film practices centered light-skinned subjects, contributing to poor rendering of darker skin Roth 2009 (the Shirley card). Evaluate capture, exposure, noise, and rendering across the people and conditions the system must serve, instead of assuming a reference face represents everyone.

Memory-color studies also examine sky and foliage. Preferred reproduction can differ from measured color, but it varies with context, observers, and the intended image. A blue swatch alone does not establish that a pixel is sky, just as a warm swatch does not establish skin. Preferred-color edits need semantic context and should preserve distinctions that matter to the photograph. There is no universal table of Lab prototypes that defines correct skin, sky, and vegetation.

A small RGB attenuation model lets us inspect the algebra behind the two-factor construction. The coefficients below are chosen for illustration. They are not measured absorption spectra, and the result is not an implementation of Tsumura et al.'s calibrated separation method. Spectral integration and scattering prevent a general identification of RGB logarithms with physical pigment densities.

import numpy as np

# --- linear sRGB (D65) -> XYZ -> CIELAB, and linear -> sRGB --------------
M_RGB2XYZ = np.array([[0.4124, 0.3576, 0.1805],
                      [0.2126, 0.7152, 0.0722],
                      [0.0193, 0.1192, 0.9505]])
WHITE = np.array([0.95047, 1.0, 1.08883])          # D65 reference white

def srgb_encode(lin):                              # linear light -> sRGB, [0,1]
    lin = np.clip(lin, 0.0, 1.0)
    return np.where(lin <= 0.0031308, 12.92*lin, 1.055*lin**(1/2.4) - 0.055)

def lin_rgb_to_lab(lin):                           # linear light -> CIELAB
    t = (M_RGB2XYZ @ np.asarray(lin, dtype=float)) / WHITE
    d = 6/29
    f = np.where(t > d**3, np.cbrt(t), t/(3*d*d) + 4/29)
    return np.array([116*f[1] - 16, 500*(f[0]-f[1]), 200*(f[1]-f[2])])

# Illustrative two-factor attenuation in RGB, not a calibrated skin model.
# These chosen vectors demonstrate log products becoming sums; their
# parameters are dimensionless controls, not measured pigment concentrations.
RGB_BASE  = np.array([0.95, 0.88, 0.80])           # chosen baseline (linear RGB)
SIGMA_MEL = np.array([0.35, 0.60, 1.00])           # melanin: absorbs blue > green > red
SIGMA_HEM = np.array([0.12, 0.85, 0.80])           # hemoglobin: absorbs green (540-580 nm) + blue

def skin_linear(melanin, hemoglobin):              # nonnegative illustrative controls
    v = np.log(RGB_BASE) - melanin*SIGMA_MEL - hemoglobin*SIGMA_HEM
    return np.clip(np.exp(v), 0.0, 1.0)            # linear-light RGB

def skin_srgb(melanin, hemoglobin):                # -> display sRGB, [0,1]
    return srgb_encode(skin_linear(melanin, hemoglobin))

def skin_lab(melanin, hemoglobin):                 # -> CIELAB (D65)
    return lin_rgb_to_lab(skin_linear(melanin, hemoglobin))

Increasing either nonnegative control reduces the linear channels according to its coefficient vector. In log RGB the two changes add; exponentiation returns to linear RGB before conversion to CIELAB or display encoding. This example shows the computation and its domains without claiming that two parameters span all skin appearance.

The other useful implementation is a color-region selector. The region must be chosen for the editing task; there is no universal CIELAB box that identifies skin, sky, or vegetation. Hue intervals can cross zero, and neutral pixels need separate handling:

import numpy as np

def in_lab_region(lab, lightness_range, hue_range, chroma_range):
    # A geometric selection in CIELAB, not an object classifier.
    L, a, b = lab
    chroma = np.hypot(a, b)
    if chroma == 0:
        return False                 # hue is undefined on the neutral axis
    hue = np.degrees(np.arctan2(b, a)) % 360
    lo, hi = (v % 360 for v in hue_range)
    inside_hue = lo <= hue <= hi if lo <= hi else hue >= lo or hue <= hi
    return (lightness_range[0] <= L <= lightness_range[1] and
            chroma_range[0] <= chroma <= chroma_range[1] and inside_hue)

# Arbitrary warm-color box for an editing experiment, not a skin-tone standard.
selected = in_lab_region([60, 20, 20], (20, 90), (20, 70), (5, 60))

A warm wall and a face can both satisfy this test. Combine a color selection with spatial or semantic evidence when the intended adjustment concerns an object. Using a geometric region for a selective edit is different from claiming that it recognizes the material.

The pigment model is useful because the two major absorbers do different things. Melanin in the epidermis absorbs broadly; increasing it tends to darken the reflected signal. Hemoglobin in dermal blood has a more structured spectrum and helps determine the reddish component and changes associated with perfusion. Light can pass through both regions and scatter before leaving the skin, so their effects combine along its paths. The simplified log model turns multiplicative attenuation into additive components, making separate controls convenient. Its usefulness does not make the fitted RGB components direct physical concentration measurements.

Subsurface transport adds a spatial effect. Light entering at one point can emerge nearby, so the color at that exit point depends on illumination over a neighborhood. A local surface-only model misses this redistribution even if its color parameters are correct. Controlled illumination in a light stage helps measure how a face responds as the incident direction changes, providing evidence for relighting models. Color calibration, pigment approximation, and spatial scattering address different parts of why the rendered face looks convincing.

Spectral sensitivity: matching the eye, catching photons, and white balance🔗⧉

With photon spectral exposure $q(\lambda)$ and effective channel detection probability $\eta_i(\lambda)$, the expected signal electrons are

The response includes filters, optics, and quantum efficiency under the stated convention. Sampled implementation needs wavelength-bin weights. Shot noise is determined by detected electrons, while read noise is added by the readout; the full model is in Chapter 2.8, Noise, signal-to-noise ratio and dynamic range. A broad response alone is not a color measurement: distinct response shapes supply the information that separates spectra.

The exact shape of a color sensor's three spectral-sensitivity curves (the $r(\lambda)$ of the red, green, and blue channels, built earlier by the color-filter array on the photosites of Chapter 2.7, Sensors: photosites, CCD vs CMOS) is one of the camera's most constrained choices, because it is pulled in three directions at once that cannot all be satisfied.

• Fidelity to human vision. For a camera to record the color a human would see, its three curves should be a fixed linear combination of the eye's cone fundamentals (equivalently, of the CIE color-matching functions), the Luther–Ives condition. A sensor that met it could be mapped to human color by a single $3\times3$ matrix with no error. Practical camera sensitivities generally only approximate it, because silicon-plus-dye curves are simply the wrong shape, and the penalty is a metameric error no color matrix can remove: two spectra that look identical to the eye read as different RGB, and two that look different can read the same. The color-correction matrix in Chapter 8.10, Auto-exposure and auto white balance is a least-squares compromise: right on average, wrong on the hard cases (saturated reds, narrow-band LEDs, foliage).

• Quantum efficiency and noise. Broad transmission bands can increase total photon collection; this is distinct from quantum efficiency at a specified wavelength: the wider each filter, the more photons reach the well and the higher the SNR. Strong overlap can make color separation poorly conditioned, because separating similar channel responses can require large signed coefficients that amplify measurement noise (compare the distinct physiological opponent processing in Chapter 3.2, Perceptual color and trichromatic vision). Narrow, well-separated filters give clean color but throw photons away and read noisier. The green-doubled Bayer is one point on this curve: spend photons and resolution where the eye cares most (luminance), economize on chroma.

• White balance across illuminants. Because each channel integrates illuminant × reflectance × sensitivity, changing the light changes the raw RGB of a fixed object. Undoing that, white balance often approximates the change with per-channel gains. Matching cone sensitivities does not make this diagonal relighting model exact; a response set that matches human colorimetry well need not be the best basis for diagonal adaptation. Good spectral design makes white balance more reliable across illuminants (→ Chapter 8.10, Auto-exposure and auto white balance).

These three pull against each other: the curves that best match the eye are broad and overlapping (bad for separation), the curves that separate color cleanly are narrow (bad for QE and for matching the eye), and keeping white balance stable across odd illuminants wants yet another shape. Every color sensor is a negotiated settlement among them (Figure 4.6.24⛶ fullscreen).

Color management, ICC, and industry standards🔗⧉

ICC profiles describe transforms between device or encoding values and a profile connection space (PCS). In the conventional ICC v2/v4 architecture, that space uses D50-referenced XYZ or Lab. A color-management module combines source and destination transforms. Profiles can use matrices and curves or multidimensional lookup tables; a printer is not generally one three-by-three matrix away from RGB.

Assigning a profile declares the meaning of unchanged values. Converting profiles changes values to meet a color-reproduction objective. For example, the triple $(0.8,0.2,0.1)$ denotes different colors in sRGB and Adobe RGB. Assigning Adobe RGB to sRGB data leaves that triple unchanged and changes its interpretation. A conversion starts with the sRGB interpretation and computes destination values for the intended match, subject to the destination gamut and rendering intent. Assign the known source interpretation before conversion. For an untagged image of uncertain origin, sRGB is a common fallback assumption, not recovered metadata.

Black-point compensation is a separate option that maps tonal ranges between different black points, commonly used with relative colorimetric conversion. Compare it with the selected profiles instead of treating it as another rendering intent. See the ICC introduction and black-point compensation white paper.

Profile construction and connection account for chromatic adaptation. Bradford is widely used in this architecture Bradford chromatic-adaptation transform, but do not prepend an extra adaptation to an already specified profile conversion. The complete Bradford formula and its white-to-white checks are in Chapter 4.2, White balance and white point. A camera’s raw gains are not simply that same XYZ transform.

For printing, use the profile matching the actual printer, ink, paper, and driver settings. If the application converts to printer values, configure the driver accordingly; if the driver owns conversion, supply the input it expects. A printer profile should not be embedded as though it described an ordinary web RGB image.

Basic photography: exposure settings (shutter, aperture, and ISO)🔗⧉

For an unchanged scene and lens transmission, optical exposure scales as

Twice the integration time collects twice the light. Multiplying the f-number by $\sqrt{2}$ halves it. Thus f/2.8 at 1/250 s and f/4 at 1/125 s give approximately the same optical exposure, but the longer integration blurs motion more and the smaller aperture increases depth of field.

A stop is the base-two logarithm of a light ratio. The setting-based exposure value uses time in seconds:

Increasing this EV by one halves optical exposure. Positive exposure compensation is a different convention: it asks an automatic mode for a brighter result relative to its metered choice. In aperture priority at fixed ISO, +1 stop will ordinarily double time. In manual mode with Auto ISO it may instead double gain while leaving the light collected unchanged.

ISO is not a third optical control. At fixed aperture and time, increasing gain scales the signal and its shot noise together. It can reduce the importance of noise added downstream of that gain, or select a sensor conversion-gain mode with lower input-referred read noise. It can also reduce highlight headroom. On an ISO-invariant sensor over the relevant range, low-ISO raw data brightened later can be close to a higher-ISO capture. The comparison in Chapter 2.8, Noise, signal-to-noise ratio and dynamic range applies only after matching output brightness and checking clipping.

The common association between high ISO and noise usually includes a reduction in collected light: we raise ISO because the scene is dim or the shutter must be fast. Add light, open the aperture or lengthen time when the photograph permits it. Raising ISO alone does not improve photon statistics.

Reflected-light metering measures light coming from the scene; it cannot separate a dark object in strong light from a pale object in weak light without additional information. Spot metering makes the chosen patch explicit. Evaluative metering uses more of the frame and may use recognized subjects. A gray card is a useful reference under the subject's illumination, not proof that a meter assumes exactly 18% reflectance. Snow and dark clothing are good tests of when compensation is needed.

Expose to the right means collecting more light while retaining the highlights that matter. It is useful when time, aperture and saturation allow it; it is not a requirement to force every histogram against the edge. A longer exposure that blurs the subject is a poor trade for cleaner shadows. Inspect individual channels when possible, because a colored light can saturate one channel before the luminance preview looks white. Keep a margin when the camera's warnings refer to a processed preview rather than raw data.

The f-number normalizes the physical opening by focal length. At the same f-number, a longer lens has a proportionally larger entrance pupil. For an extended scene of equal radiance, that compensates for the change in image scale so the on-axis irradiance is approximately unchanged, apart from transmission and close-focus effects. Aperture diameter alone would not give that useful comparison across focal lengths.

The familiar series f/1.4, f/2, f/2.8, f/4, f/5.6, f/8, f/11, f/16 follows from area. Each full-stop step multiplies the f-number by about the square root of two and therefore halves optical exposure at fixed time. Doubling time compensates for that step. For example, f/4 at 1/250 s and f/5.6 at 1/125 s collect approximately the same light per sensor area, with different motion blur and depth of field. Equal collected light does not make them interchangeable photographs.

Metering: from printed guides to the sensor🔗⧉

Before built-in meters, exposure tables related film speed to illumination and subject conditions. A handheld incident meter measures light arriving at the subject, while a reflected meter measures light leaving a selected area. Through-the-lens metering accounts for some effects of the actual optical path. None of these devices knows which part of a photograph we intend to emphasize.

For repeatable work, meter a defined patch or use a controlled reference under the same illumination. Record compensation and whether settings were locked. Absolute photometry requires more than the settings printed in EXIF; Chapter 17.2, Camera response and radiance calibration develops that distinction.

A reflected reading changes when a gray card is replaced by a white card under the same light. If an automatic system treats the brighter reading as a reason to shorten exposure, it can render the white card too dark. An incident reading taken at the subject responds to the illumination instead of that change of card reflectance. The white card can then remain lighter than the gray one under the same exposure. This is the practical attraction of an incident meter in a controlled setup; it does not determine the photographer's desired rendering.

The cost is access. You can aim a reflected-light spot meter at a distant subject from the camera position. Taking an incident reading requires sampling the light at a representative subject position and choosing how to orient the receptor. Mixed lighting and directional sources make that choice consequential. Meter type changes which quantity is measured, not whether judgment is needed (Figure 5.2.12⛶ fullscreen).

Printed exposure guides made the first choice without measuring the scene. Film packaging and exposure calculators listed settings for bright sun, overcast light, and shade. The sunny-16 rule summarizes one such starting point: in bright sun, use f/16 and a time near the reciprocal of the film-speed number, about 1/100 s for ISO 100. It is a lighting estimate, not a measurement of every subject or a guarantee against clipped highlights.

Photocells made the estimate responsive to actual light. A separate or body-mounted meter could still see a different field from the taking lens and miss the loss from an added filter. Through-the-lens metering brought the measurement into the optical path that forms the photograph, accounting for transmitted light. Sensor-based metering brings it closer still: the camera analyzes a live readout from the imaging sensor. Preview processing and capture settings can differ, but the remaining decision, which tones matter and where to place them, still requires an inference about the scene and the photographer's intent.

Indirect illumination and bounce flash🔗⧉

Direct on-camera flash can produce a small-source shadow and a strong frontal reflection, which may be exactly the desired appearance. For softer directional light, bounce off a suitable wall or ceiling. The illuminated area becomes the source seen by the subject. Its location, apparent size, color and distance matter; a distant dark ceiling can consume most of the available flash energy.

A small diffusion cap on a flash does little to enlarge the apparent source outdoors. Indoors it may spread light onto walls that then become larger sources. A softbox or scrim works through the illuminated area visible from the subject, so bringing it closer generally increases its angular size.

A white reflector redirects available light into shadows. A silver surface is more directional, and a colored surface tints the reflected illumination. A black panel removes fill. Move the panel while watching the face before adding another lamp.

Red-eye is more likely when the source lies close to the viewing axis and the pupil is large. Moving the flash away from the lens axis or changing the lighting geometry can reduce the return from the eye. Software can alter its appearance afterward, but it does not change the lighting recorded on the rest of the face. The geometry is a retroreflection: an on-camera flash a centimeter or two from the lens sends light through the pupil, the retina and the blood-rich choroid behind it return it red, and it comes back along nearly the same line into the lens. Moving the source off axis throws that return cone clear of the lens, and a pre-flash that contracts the pupil shrinks it. Many animals show green or white eyeshine instead, from the reflective tapetum behind their retinas (Chapter 3.6, Animal eyes).

The digital darkroom🔗⧉

A parametric developer stores a recipe applied to source data. A layered editor stores image elements, masks and operations in a document. These approaches overlap: developers have local masks and retouching, and layered editors can retain reversible adjustment layers and embedded raw sources. The useful distinction is what can be revised and reproduced, not whether an edit is global or local.

Begin by selecting a frame whose expression, focus and motion are suitable. Check highlight clipping and the darkest important detail before spending time on polish. Set white balance for the intended rendering, correct geometry when needed, then establish overall lightness and contrast. Local adjustments can separate the subject or control a bright window without forcing the rest of the frame to follow the same curve. Revisit earlier choices when they interact.

Noise reduction and sharpening should support the intended output. Strong denoising can erase fine structure; sharpening can emphasize both texture and noise. A lens profile can correct systematic geometry or falloff, but resampling changes sampling density and brightening corners also amplifies their noise. “Correction” is still an operation with a cost.

Save a master with enough precision and a documented color interpretation. Export a separate version for the destination, including its color profile, dimensions and output sharpening. If masks depend on an external model or application cache, preserve those dependencies or save the rendered mask. The same recipe numbers can produce different pixels under another processing version.

Chapter 8.3, Tone point operations and Chapter 18.2, Global tone mapping explain tonal controls, Part 15, Demosaicking and joint methods explains reconstruction from a color mosaic, and Chapter 8.13, A raw developer in the browser provides a working developer. An exposure slider changes the recorded values; it cannot add photons or recover a fully clipped channel.

A parametric edit stores a recipe alongside the source rather than repeatedly replacing its pixels. For example, the recipe may record an exposure adjustment, a crop rectangle, and a local mask. Changing the crop later re-renders from the source instead of enlarging an already cropped export. Applying a common white-balance or rendering adjustment to a series likewise changes the recipes while preserving the captures.

This is different from claiming that every operation is reversible in an exported image. A JPEG export may clip highlights, quantize tones, and discard material outside the crop. The editable state survives because the source and recipe survive, not because the export contains enough information to undo every decision. Pixel layers, retouching strokes, and generated content may add further data that must also be retained.

A developer also helps manage a shoot: importing, comparing frames, rating, keywording, and applying a common recipe to a series. A layered editor is useful when the document must keep separate image elements, retouching, masks, or text. Many workflows use both. These are useful emphases, even though modern tools overlap and neither category is restricted to global or irreversible edits.

The control panel is a list of computational problems:

The names do not uniquely specify an implementation. A healing tool need not use the same algorithm as another application's healing tool. Learning the underlying problem lets us ask what information a control changes, which assumptions it makes, and which artifacts to inspect.

The contrast is limited: dynamic range and gamut🔗⧉

Two output constraints are easy to confuse. A print cannot emit the luminance of a sunlit scene, and its ratio between darkest and lightest regions may also be smaller. Scaling all values addresses the first mismatch but not the second. For a sense of scale, a sunlit scene with deep shade can span ten thousand to one or more, while a print under room light spans a few hundred to one at best. Tone mapping, local adjustments and lighting choices affect which distinctions survive.

A bright source can be suggested by a glow or surrounding dark tones even when the display cannot reproduce its luminance. That is a perceptual representation, not an absolute photometric measurement. Likewise, gamut mapping chooses a reproducible color when the target cannot display the original one. Chapter 18.10, HDR displays and display mapping and Chapter 4.7, Synthesizing color describe the output limits; Chapter 5.5, Developing and editing a photograph shows the photographer's choices within them.

The history of depiction makes the intensity mismatch tangible. The lecture contrasts a painted sky with Brunelleschi's use of a reflecting silver surface for the sky in his perspective demonstration. A pigment and a reflection have different access to the surrounding illumination. Neither comparison can be reduced to an image's normalized pixel range.

The Zone System coordinates exposure, negative development and printing around planned tonal placements Adams, The Negative. Dodging and burning modify those placements locally. Characteristic curves compress the ends of the range, while perceptual tone-reproduction methods can also model visibility, glare or adaptation Ferwerda et al. 1996. These are distinct responses to limited output, not interchangeable names for a gamma curve. Retinex provides a related account of appearance through spatial relationships Land & McCann 1971, but it should not be read as proof that the eye simultaneously measures the entire luminance span across all environments.

A blur zoo🔗⧉

With the mechanics settled, the only choice left is the kernel's shape. Two are worth knowing.

The box filter gives every sample in a $(2r+1)\times(2r+1)$ window weight $1/(2r+1)^2$. Its square footprint is not rotationally symmetric, and its frequency response has lobes and zeros. However, its nonnegative weights cannot create overshoot beyond the neighborhood’s range; a blurred step becomes a ramp, not a ringing edge. Running sums can make large box averages inexpensive.

The Gaussian is the workhorse. Its weights fall off smoothly with distance from the center, following the bell curve

where $r$ is the distance to the center and $\sigma$ (the standard deviation) sets the width, the blur's "radius" in spirit. Read it back: a neighbor's weight decays exponentially with the square of its distance, so nearby pixels count for a lot and far ones fade smoothly to nothing. The Gaussian is smooth, rotationally symmetric (no directional bias), and, as the next chapter shows, uniquely well-behaved in frequency, which is why it is the default blur almost everywhere.

Run both on an actual photograph and the difference in feel is plain: the box smears fine texture into faintly blocky, slightly directional mush, while the Gaussian dissolves it smoothly and isotropically. Use a Gaussian when the blur should look natural (Figure 6.2.13⛶ fullscreen).

Better than reading about them, you can try the blurs directly (Figure 6.2.14⛶ fullscreen): switch kernels, change the width, and watch what each does to real texture.

An ideal Gaussian has infinite support. At a distance $3\sigma$, its weight is $e^{-9/2}\approx0.011$ of the peak; a one-dimensional cut at $\pm3\sigma$ discards about 0.27% of the mass. Choose support for the required accuracy and renormalize the sampled, truncated weights. Square truncation preserves separability; circular truncation generally does not. Sampling and truncation make rotational symmetry approximate.

To simulate physical mixing, decode to linear light, filter, then encode for display. Aesthetic smoothing can deliberately use another space, but it should be named. Linear-light processing is essential when the blur is part of a calibrated forward model. For images with transparency, filter premultiplied color and alpha consistently to avoid color bleeding from transparent pixels.

The visible encoding error is easiest to see where black meets white. Averaging their encoded values gives a middle code value; decoding that value generally yields less light than averaging the original linear intensities. A blurred highlight can therefore acquire a dark transition when filtering is done in the display encoding. The arithmetic is still linear in the stored numbers, but it no longer represents linear mixing of light.

Domain operations: moving pixels around🔗⧉

Recall the three families of image operation from Chapter 8.3, Tone point operations, sorted by what each output pixel is allowed to read. Resampling is the domain family: the underlying continuous image is remapped in position; discrete resampling generally blends its recorded colors. Formally a domain transform applies a function $f$ from the plane to itself (from $\mathbb{R}^2$ to $\mathbb{R}^2$) to the image's coordinates, with the rule that if the point $(x,y)$ carried color $c$ in the input, then $f(x,y)$ carries that same color $c$ in the output. The range (the colors) is untouched; only the domain (the coordinates) is remapped.

That one definition spans a huge amount of ground. When $f$ is a uniform scale it is resizing; a rotation straightens a horizon; an affine or perspective $f$ fixes keystoning or stitches a panorama; and a freely chosen $f$ is a warp: picture the image printed on rubber and stretched. The free-form case, including how a user even specifies such an $f$ (and the morphing it unlocks), gets its own Chapter 11.2, Antialiasing for complex transforms. Here we take $f$ as given and concentrate on the hard part shared by every domain transform: the new positions almost never land on the input grid, so we must invent values between the samples.

Other forms of linearity, and multilinearity🔗⧉

We have treated $A$ as a linear operator on the scene radiance $\mathbf{x}$, taking the illumination as given. But radiance is not the only quantity imaging is linear in. Light adds and scales, so the image is equally linear in the illumination: double every source and every pixel doubles; light a scene by the sum of two illuminants and you get the sum of the two images. That superposition is what makes light stages, photometric stereo, and relighting work.

The image is linear in still other quantities: for a fixed scene a convolutional blur is linear in its kernel exactly as it is linear in the image (the fact that makes blind deblurring so slippery), and a pixel is linear in the sensor's spectral response. Each is a different axis along which the same measurement is a linear map.

The complication is that these linearities usually multiply. A pixel is the product of what the surface reflects and what the light delivers; a blurred image is the product (convolution) of a sharp image and a kernel. Such a measurement is bilinear: linear in each factor with the other held fixed, but not jointly, and more generally multilinear when several unknowns combine. Joint least-squares objectives for bilinear models are generally non-convex even though each factor alone is linear, so when both factors are unknown we cannot simply invert a matrix; we alternate between them, or factor the product apart with priors. Everything earlier in this chapter (conditioning, invertibility, the reliance on priors) carries straight over, now doubled.

Color constancy separates illuminant spectrum and surface reflectance inside the sensor’s wavelength integral. That integral is bilinear in the two spectra when the other transport factors are fixed. A channel-wise RGB product is only an approximation, as discussed in Chapter 4.2, White balance and white point. The ambiguity is visible even in a scalar product: multiplying one factor by a constant and dividing the other by it leaves the measurement unchanged. Priors or calibration must constrain that scale and other ambiguities.

Brainard and Freeman used a Bayesian formulation to reason about color constancy, while Tenenbaum and Freeman used bilinear models to separate style from content. The recurring issue is how observations constrain two interacting factors. Gray-world assumptions, reflectance statistics and calibration constrain different aspects of the ambiguity; merely naming both factors does not make them separately observable. Chapter 7.1, Imaging as Bayesian inference with probabilistic priors develops how such assumptions enter an estimate.

Harder inverse problems: factorization🔗⧉

Everything above assumed the forward model is linear: a known operator $A$ acting on one unknown. A whole second family of problems drops that, and because it recurs constantly it deserves a name before we leave the chapter. In these the measurement is a product of two unknowns, and we must recover both factors from their product alone.

The canonical version is pure arithmetic. Hand someone the number 12 and ask which two numbers were multiplied to make it. The answer is radically underdetermined: $3\times4$, $2\times6$, $1\times12$, $2.4\times5$ all fit, and nothing in the product alone prefers one. Only extra knowledge (a prior on what each factor is allowed to be) breaks the tie. That is the same logical move as regularization above, now doing even heavier lifting.

In the fully general case, factorization is our familiar linear problem $\mathbf{y}=A\mathbf{x}+\mathbf{n}$ with the ground shifted beneath it: now $A$ is unknown too, and we must recover the operator and the scene it acted on from their product alone. That extra unknown is what tips these problems from merely ill-conditioned to ambiguous.

A striking number of imaging problems are exactly this shape:

• White balance: the raw sensor value is (to first order) the illuminant times the surface reflectance: $\text{observed}=\text{light}\times\text{material}$. From the product we must separate what color the light was from what color the object is. How hard the factorization is depends on how much we let the light vary. Global white balance assumes one illuminant for the whole frame, so a single triplet of coefficients (three numbers) explains every pixel: there are fewer illumination parameters, but unknown reflectances still leave an ambiguity. Local (spatially varying) white balance lets the light change across the scene, so each pixel becomes its own independent product of two color triplets, the local light times the local material. That is far more ambiguous, and it takes a strong prior on how illumination varies to pin it down (Figure 4.2.3⛶ fullscreen) [@hsu-etal-2008; @boyadzhiev-etal-2012].
• Intrinsic images: an image is shading times reflectance (albedo), $I=S\times R$: the geometry-and-illumination term times the material term. Split them and you can relight, recolor, or lift shadows.
• Matting: a composited pixel is $\alpha F+(1-\alpha)B$: foreground color, background color, and opacity $\alpha$, coupled multiplicatively into one observed color (the equation of Compositing, segmentation and matting).
• Blind deblurring: the blurry photo is the sharp image multiplied by an unknown blur matrix (blurring is a linear operator, so the observation is the product of that operator with the image) and we know neither the image nor the blur (Chapter 16.4, Blind deblurring).

These factorization (or blind, or separation) problems share two features. First, they are even more ill-posed than a linear inverse problem, because an unconstrained model can admit a trivial split: dump everything into one factor and leave the other the identity (the "$1\times12$" answer, which for blind deblurring is precisely the "no blur, the image is already sharp" trap). Additional constraints, calibration, or priors (illuminants are few and smooth, reflectances piecewise-constant, natural images sparse-gradiented, mattes nearly binary) can reduce ambiguity without guaranteeing uniqueness. Second, the forward model is no longer linear, but it is usually _multilinear_: linear in each factor separately while the others are held fixed ($\alpha F$ is linear in $F$ for fixed $\alpha$, and linear in $\alpha$ for fixed $F$; $S\times R$ likewise). That structure is exactly what keeps them tractable: it invites alternating schemes (freeze one factor, solve the other as an ordinary linear problem, alternate), which is why so many of these methods read as coordinate descent. Seen this way, white balance, intrinsic images, matting, and blind deblurring are the same structural problem in four different applications: recover the factors from their product, and use a prior to choose among the many that fit.

Stochastic quantization and dithering🔗⧉

Stochastic rounding chooses between adjacent levels. For an input $v$ between $q_0$ and $q_1$, set $p=(v-q_0)/(q_1-q_0)$ and choose $q_1$ with probability $p$, otherwise $q_0$. Then $\mathbb E[q]=v$. This unbiasedness assumes no clipping and is expressed in the units used to define the levels. Independent decisions introduce noise with variance $p(1-p)(q_1-q_0)^2$ Gupta et al. 2015 (stochastic rounding).

Error diffusion passes each quantization residual to unprocessed neighbors. Floyd–Steinberg distributes it using weights $7/16,3/16,5/16,1/16$ in a raster scan Floyd & Steinberg 1976. Ostromoukhov varies the coefficients with intensity Ostromoukhov 2001. These methods shape the error rather than making it vanish, and can produce directional or periodic patterns.

To preserve average emitted light, use the actual linear-light values of the available output levels. For gamma-encoded levels, decode the two neighboring codes before calculating the rounding probability or residual. Simply dithering uniformly in code space preserves average code value, not average light. Boundary error loss and clipping can also bias error diffusion.

Capture precision and editing precision serve different purposes. Raw files often retain finer quantization and more highlight information than a rendered 8-bit image, but additional bits cannot recover a saturated photosite. A highlight clipped only by the rendering may remain available in raw. Sixteen-bit integers offer fine uniform steps over a fixed range; half floats offer a wider range with less significand precision. Choose the representation for the arithmetic and range, rather than treating all 16-bit formats as equivalent.

Error diffusion gives the rounding a memory. If one pixel is rounded upward, its positive quantization error is subtracted from later inputs so that neighboring pixels can compensate. The resulting pattern trades spatial texture for a better local average. The average being preserved must be specified: averaging linear-light samples predicts emitted light, while averaging encoded samples preserves a different quantity. This matters for a one-bit projector or mask as well as for a displayed photograph. A locally correct average does not make the pattern invisible; viewing scale and the spatial distribution of error still matter.

For a value 70% of the way between two available levels, stochastic rounding chooses the upper level 70% of the time. Across many independently rounded pixels, the average approaches the desired value even though each individual pixel is quantized. At a viewing scale that blends the dots, this can replace a visible band with a textured tone. The same idea applies to a binary light mask or a projector with few available intensities: the spatial mixture approximates an intermediate response.

Pixel coordinate conventions🔗⧉

Before we trust ourselves to index an image, two small geometric conventions have to be nailed down, because both are silent traps and both differ between tools.

The first is where the origin sits. We place $(0, 0)$ at the top-left pixel, with $x$ increasing to the right and $y$ increasing downward: the order in which image data is scanned and stored, and the convention most imaging code (and our own camera frame) uses. The catch: a few systems (classic OpenGL textures, and much plotting and math software) put the origin at the bottom-left, with $y$ increasing upward, as on a graph. Hand an image from one world to the other without flipping it and the picture comes out upside down (Figure 8.1.10⛶ fullscreen). Nothing crashes; it merely looks wrong, which is the worst kind of bug to chase.

We assign integer coordinates to pixel centers. A displayed pixel cell at $(0,0)$ spans $[-0.5,0.5)$ along each axis, and the image footprint spans $[-0.5,W-0.5)\times[-0.5,H-0.5)$. This geometric cell convention does not assert a box-shaped sensor response. APIs that place integer coordinates at cell corners differ by half a pixel.

That half-pixel sounds pedantic, and for a point operation it is: a curve applied to every pixel never asks where a pixel is. But the moment you resample or warp (scale up, rotate, correct lens distortion) you are evaluating the image at non-integer positions, and a consistent half-pixel error shifts the whole result by half a pixel, blurs it, or makes a "round trip" (shrink then enlarge) fail to land back where it started. It is one of the most common bugs in resampling code precisely because it never raises an error. We fix the convention now (integer = center) and return to it in full at resampling and warping, where it earns its own discussion of "where-from vs. where-to" addressing.

Video and more dimensions🔗⧉

Nothing forces us to stop at a flat grid, either. A video is just an image sequence with one extra axis, time: $H \times W \times C \times T$. Most operations we write for a single frame generalize directly (run them frame by frame) and only the operations that couple frames (motion estimation, temporal denoising) need anything new, which we save for the motion and video material.

The same move works in other directions. Volumetric imaging stacks 2-D slices into a 3-D block: medical computed tomography (CT) and magnetic resonance imaging (MRI) produce an $H \times W \times D$ volume. A light field adds two angular axes to the two spatial ones (a 4-D record of rays, not just pixels); a spectral cube adds wavelength. In every case it is the same array machinery with more axes. We write this book for 2-D images and point out where adding a dimension is all it takes.

Pixel lookups and boundaries🔗⧉

Almost every operation in the coming chapters reaches for a pixel's neighbors, which forces an unavoidable question: what is $I(x, y)$ when $(x, y)$ lands outside the image? We cannot dodge it, because blurs, gradients, resampling, and warps all ask for out-of-bounds pixels right at the border, where there are no neighbors on one side. Reading past the array is, at best, garbage and, at worst, a crash, so the first discipline is defensive: route every pixel read through a single accessor function, a small get(x, y, c), so that out-of-bounds access is decided in exactly one place rather than re-improvised (and re-broken) in every loop. As a bonus, that one function is also where the memory layout, the channel order, and the coordinate convention live, so the rest of the code can think in plain $(x, y, c)$ and forget every packing decision above. It is the natural home for one more safety check, too: asserting that an index lies in $[0, \text{size})$ inside that single function turns a distant, hard-to-trace crash into an immediate, localized error (we develop this habit in the debugging chapter).

The accessor then has to decide what a pixel past the boundary means. There are four standard policies (Figure 8.1.11⛶ fullscreen):

• black / zero: treat everything outside as $0$. Simple, but it darkens edges, since you are averaging in black that is not really there.
• clamp / replicate: repeat the nearest edge pixel outward. A reasonable, neutral default for many operations.
• mirror / reflect: fold the image back across its boundary, so the neighborhood just past the edge mirrors the one just inside. This avoids both the artificial darkening of zero and the streaking of clamp, and is the usual safe default for filtering.
• wrap / tile: treat the image as periodic, the left edge adjacent to the right. Rarely what you want visually, but it is exactly the assumption the Fourier transform makes, so it returns when we get there.

On a real photograph the four policies look like Figure 8.1.12⛶ fullscreen: the same crop, continued past its border each way, surrounded by a dark frame, streaked outward, folded back on itself, or wrapped around from the opposite edge.

The boundary rule changes filtering and resampling near the border. We use reflection for these examples, stating whether the edge sample is repeated: libraries distinguish whole-sample and half-sample reflection. Neither reconstructs unseen content. For inverse problems, the adjoint must include the selected extension rule; see Chapter 6.9, Efficient linear solvers.

What the numbers actually mean: color spaces🔗⧉

Those three letters, "RGB," hide more than they reveal. Several independent decisions all travel under the same name, which is why two arrays can both be rightly called "RGB" and still be incompatible. The decisions that vary:

A bare array announces none of this: the information lives in metadata (the file header, an embedded ICC color profile). The advice is correspondingly simple: read the metadata, don't assume. When two images that "should" match stubbornly don't, this table is the first place to look.

Contrast: steepen about a pivot🔗⧉

Where exposure moves all the tones the same way, contrast spreads them apart or pulls them together. The idea is to pick a pivot value, a mid-gray that stays put, and push every other value away from it (more contrast) or toward it (less). Values above the pivot get brighter, values below get darker, and the pivot itself never moves. As a curve, that is a straight line through the pivot whose slope is the contrast gain g:

where p is the pivot. Read it as: measure how far a value sits from the pivot, scale that distance by the gain, add it back. A gain above 1 steepens the line and raises contrast; a gain below 1 flattens it and lowers contrast. The steeper the line, the sooner it slams into the floor at 0 and the ceiling at 1, so strong contrast clips both shadows and highlights (Figure 8.3.5⛶ fullscreen).

The pivot and the domain both affect contrast. A linear-domain adjustment preserves its chosen linear pivot; a log-domain scale changes luminance ratios by a power. An encoded-domain adjustment acts on displayed code differences. Choose the appearance you want and inspect the clipping at both ends (Figure 8.3.6⛶ fullscreen). On the photographs in the figure, the three give recognizably different looks. Steepening the encoded values, what most editors do, pivots about a perceptual mid-gray and roughly tracks how the eye judges contrast. Steepening linear values weights the highlights more and, on these images, reads harsher, with shadows crushed faster. Steepening log values stretches ratios uniformly and reads gentlest across a wide range. None is wrong; they are three looks, and an editor can offer the choice.

Black point, white point, and levels🔗⧉

Levels maps chosen input endpoints a and b to output black and white. For $b>a$, a simple version is

The midpoint parameter $\gamma_m$ changes the interior without moving the endpoints. Handle coincident endpoints explicitly. Filling the whole output range is a choice: a foggy or low-contrast scene need not contain a true black or white.

Subtracting a calibrated additive light offset is meaningful in linear light. General haze also attenuates scene light and varies with depth, so an arbitrary black-point adjustment is not a complete haze inversion. A histogram minimum cannot identify the offset without further assumptions; see Chapter 8.2, What the numbers mean.

A display-space levels adjustment is an appearance operation. A sensor black-level correction is calibration in raw units. Keep them separate even if both lower dark values.

Use the histogram to see which values an endpoint choice will clip, then judge the image. Small percentile trims can avoid letting a single outlier set the range, but they still intentionally discard extremes (Figure 8.3.7⛶ fullscreen).

An additive pedestal can explain a washed-out image. Add the same positive value to a dark patch and a bright patch: their difference stays the same, but their ratio moves closer to one. Subtracting a known pedestal in linear light reverses that part of the change. Choosing a black point by eye is less specific; it can improve a rendering without estimating the physical stray light. If the scene really has no black object, forcing its minimum to zero removes valid signal. Haze adds the further complication of depth-dependent attenuation as well as scattered light.

Local point operations🔗⧉

A fixed mask m allows a spatially varying point adjustment:

This blends the original and adjusted values in the specified working space. For exposure, blending the two outputs gives a factor $1-m+m2^s$; scaling the stops by the mask instead gives $2^{ms}$. They agree at mask values zero and one but differ in between. A mask inferred from neighboring pixels makes the complete algorithm nonlocal, even though applying the finished mask is pointwise.

The connection to the darkroom is exact (Figure 8.3.10⛶ fullscreen). Wherever the mask is bright, the adjustment acts; where it is dark, the pixel is untouched. Run a local exposure through a mask and you have reinvented dodging (locally brightening, holding light back in the printing easel) and burning (locally darkening, giving extra exposure): the only difference is that the mask is now a soft digital matte instead of a piece of cardboard waved over the paper. A radial gradient lifts a face out of the shadows; a linear gradient is the graduated-ND filter that holds back a bright sky; a painted brush burns down one distracting highlight. Everything earlier in the chapter, exposure, contrast, the tone curve, becomes a local tool the moment it is gated by a mask.

Selections can come from a value or color threshold, a painted brush, a gradient, an edge-aware tool, or semantic segmentation. These choices change what the complete operation depends on. An edge-aware selection uses image structure to keep an adjustment within an object; its use of color similarity connects it to the affinities used by the bilateral filter. The finished mask can still be applied with the pointwise blend above.

Basic color enhancement: saturation and vibrance🔗⧉

A simple saturation control moves colors toward or away from a gray of equal chosen brightness. The definition of brightness matters: preserving an encoded weighted sum does not preserve physical luminance or perceived lightness.

That split is literally one line of code. Let $Y$ be the pixel's luma: a weighted sum

computed here on the gamma-encoded $R,G,B$ (so it is a luma $Y'$, not a linear luminance) and broadcast back to all three channels; then its chrominance is just $\text{RGB} - Y$ (what is left after the gray is removed), and saturation scales it:

The weights sum to one, so this operation preserves the same weighted channel sum before clipping. At s=0 it gives gray, at s=1 the input, and at s>1 it extrapolates away from gray. Here we use sRGB-encoded values as a simple appearance control. The same construction in linear RGB preserves luminance before clipping; a perceptual-space chroma edit has different behavior.

Vibrance is a gentler cousin: it boosts the saturation of muted colors more than already-vivid ones, and typically spares skin tones, so portraits do not turn garish (Figure 8.5.2⛶ fullscreen). It is the same formula, but with the constant $s$ replaced by a per-pixel gain that is large where a pixel is dull and fades to nothing where it is already colorful (and near skin hues):

For this example, use HSV saturation in $[0,1]$, with zero at black. A chroma-to-luminance ratio is not automatically in that range and cannot be substituted without normalization. The optional hue weight reduces changes near selected orange hues, not verified skin. It can affect wood and fabric too. A neutral pixel stays neutral because its chroma is zero, even where the nominal gain is largest.

HSV is a convenient coordinate system for this teaching implementation; commercial vibrance controls need not use it. For nonnegative RGB, with M=max(R,G,B), m=min(R,G,B), and C=M−m:

One caveat: "how far from gray" and "leave lightness alone" both depend on the color space you measure them in, and doing this well means scaling chrominance in a perceptually uniform space rather than naively in RGB. That choice of space, and why skin needs protecting at all, is the subject of the color part, so it is what we defer to Chapter 4.1, Linear color reproduction and CIE XYZ; the mechanics are the two equations above. Both saturation and vibrance are color operations, not tone operations.

For C>0, the HSV coordinates are

At C=0, hue is undefined; an implementation can store zero by convention and avoid evaluating the divisions by C. Value is the largest channel, and saturation is the channel range relative to that value. HSL instead uses $L=(M+m)/2$ as its lightness coordinate. Neither this L nor HSV value is physical luminance. These coordinates are inexpensive controls for selecting and changing colors, not perceptually uniform measurements.

Any space that separates a lightness from a chroma supports the same saturation formula, and the choice of space changes the result (Figure 8.5.3⛶ fullscreen).

Histograms🔗⧉

A histogram helps judge exposure when glare makes the camera preview hard to read. It shows how many pixels occupy each part of the value range. First check what it measures: most camera histograms describe the rendered preview, so their clipping warnings need not coincide with clipping in the raw sensor data.

The histogram🔗⧉

The histogram of an image is the distribution of its pixel values: for each value (or, in practice, each small bin of values) how many pixels carry it (Figure 8.7.1⛶ fullscreen). That is the whole definition. You sweep the value axis from black (0) to white (1), and at each point you plot a count. A grayscale image gives a single curve; a color image gives three, one per channel, usually drawn overlaid.

A histogram discards position. Shuffle all the pixels and it stays unchanged. This makes it a useful summary of tones, but it cannot tell whether a bright patch is a disposable reflection or an important face. Read it alongside the photograph.

Counts at an endpoint are a reason to inspect that end of the range. They may indicate clipping, but a black background or a uniform white graphic can also produce a spike. An empty part of the range is not necessarily wasted: fog and low-contrast subjects need not contain black and white. Check individual RGB channels as well as luminance, since a saturated red can clip while its luminance remains modest.

Exposure changes the distribution, but the motion depends on the horizontal axis. Multiplying linear values by four stretches their positions away from zero; adding two stops translates log values by two. Reducing a floating-point image by two stops does not itself crush its shadows to zero. Clamping, quantization, sensor noise, and the display determine what detail survives.

A joint three-dimensional color histogram counts RGB triples, retaining channel relationships that three separate histograms discard. Two images can share all three channel histograms and still contain different colors. Joint histograms support palette analysis and color transfer, but need more samples as bin resolution grows. Their geometry also depends on the color space.

The histogram depends on the encoding space🔗⧉

The trap is simple: the shape of a histogram depends entirely on the numerical space the values are stored in. The same image (the same physical scene, the same pixels of light) has a completely different histogram in linear light than in gamma-encoded (sRGB) values than in a log encoding. Nothing about the scene changed; only the ruler we used to measure each pixel did, and the histogram is a picture of that ruler as much as of the scene (Figure 8.7.3⛶ fullscreen).

Uniform bins in linear light count equal differences in light. Uniform log bins count equal ratios; sRGB allocates more numerical range to shadows than linear encoding does. Its main power-law segment uses exponent $1/2.4$, with an offset and a linear toe. These different bins give different histogram shapes even for identical pixels.

For a normalized linear value of 0.18, sRGB gives about 0.46. A log code value cannot be specified without the log curve, scale, and offset. Always label the encoding and range of a histogram, including whether the quantity is a channel, linear luminance, or encoded luma.

A histogram is a sampled estimate🔗⧉

A histogram counts the values in the finite image exactly, subject to the chosen bins. As an estimate of an underlying scene distribution, it also reflects sampling and quantization:

Binning. You chose a finite number of bins (256 is common for 8-bit data), and the bin width sets the resolution. Too few bins smears real structure together; too many leaves the counts noisy and gappy. The comb-like ridges you sometimes see are an artifact of the bin grid, not the scene.

Quantization. If the underlying values are themselves quantized (an 8-bit image has only 256 distinct levels) then the histogram inherits spikes and gaps: tall counts at the levels that exist and empty bins between them. This is especially severe after a stretch (levels or equalization), which spreads 256 input levels across a wider output range and leaves visible combing. Those gaps are an artifact of the quantization, not evidence of anything about the photograph. Read the overall shape (where the mass sits, whether it clips) and do not over-interpret the fine teeth.

Histogram equalization🔗⧉

Now use it: the histogram can drive an automatic tone curve. The classic move is histogram equalization, and it rests on one clean idea. Recall the cumulative histogram, or cumulative distribution function (CDF), the running total of the histogram, swept from black to white, climbing monotonically from 0 to 1. Use that CDF as the tone curve itself:

That is the entire algorithm: build the image's CDF, then push every pixel value through it. Concretely, count the pixels into bins, sweep left to right accumulating a running total, normalize it to climb from 0 to 1, and use the result as a per-pixel look-up table:

# Finite scalar values in [0,1], nonempty image, B >= 2.
# Preserve a constant image: it has no contrast to redistribute.
if min(im) == max(im): return copy(im)
bin(v) = min(B-1, floor(B*v))
hist = zeros(B)
for v in im: hist[bin(v)] += 1
cdf = cumulative_sum(hist) / number_of_pixels(im)
for p in pixels(im): out[p] = cdf[bin(im[p])]

There are two ways to see why this works, one about ranks and one about slopes.

The CDF at a value $v$ is the fraction of pixels darker than $v$, so mapping each pixel through it replaces its value by its rank: a pixel brighter than 30% of the image goes to 0.3, one brighter than 90% goes to 0.9. The darkest tenth of the pixels therefore lands in $[0, 0.1]$, the next tenth in $[0.1, 0.2]$, and so on. Every tenth of the output range receives a tenth of the pixels, which is what a flat histogram means. The curve keeps the order of the pixels, since the CDF never decreases, and throws away everything else about their values: only how many pixels are darker matters.

The slope gives the same answer locally, and it is the version that explains what the curve does to contrast. The derivative of the CDF is the histogram itself, normalized: $\text{CDF}'(v)=h(v)$, the fraction of pixels per unit of value at $v$. A tone curve's slope is how much it stretches a small range of input values. A thin slice $[v, v+\Delta v]$ holds a fraction $h(v)\,\Delta v$ of the pixels and is stretched to an output range of width $h(v)\,\Delta v$, the same number. Output range is handed out in proportion to how many pixels need it. Where pixels crowd, the curve is steep and their small differences are stretched apart; where few pixels live, it is flat and spends almost nothing. For a continuous distribution the output density is exactly $h(v)/\text{CDF}'(v)=1$: uniform.

A small example makes the trade concrete. Suppose 80% of an image's pixels have values between 0.2 and 0.3, a dull, low-contrast scene, and the rest are spread from 0.3 to 1. The CDF climbs from 0 to 0.8 across $[0.2, 0.3]$, so that range of 0.1 is stretched to 0.8 of the output, a slope of 8: the crowded midtones gain eight times their contrast. The remaining 0.7 of the input range is squeezed into the last 0.2 of the output, a slope below 0.3, and loses contrast. Equalization spends contrast where the pixels are, which is the same budgeting argument that drives tone mapping of high-dynamic-range images (Chapter 18.2, Global tone mapping).

In Figure 8.7.4⛶ fullscreen a band of input levels is projected through the CDF onto an output interval as wide as its share of the pixels.

A finite image has ties: all pixels at one input value receive the same output, so a deterministic curve cannot split their count among several output bins. Equalization allocates more output range to crowded tones, but does not guarantee a flat discrete histogram or maximize every measure of contrast. In the worked example above with bins 0.05 wide, the two crowded bins become two spikes of 40% each, and only four of ten output slices receive any pixels: equalization stretches the range between crowded values, but cannot split a single value.

Once computed, the CDF is a point operation, conveniently stored in a LUT. Choosing that curve is image-dependent. There is a useful distinction from histogram shape: a strictly increasing encoding preserves ranks. For such an encoding $g$, $F_{g(L)}(g(L))=F_L(L)$. Ideal CDF equalization therefore gives the same output ranks before and after encoding. Finite bins, quantization, ties introduced by clipping, and the chosen output encoding can change the practical result.

Equalization has no model of subject matter or noise. A nearly uniform wall can occupy many pixels but a narrow range of values; equalization stretches that range and makes its noise conspicuous. Use it cautiously for photographic contrast.

Figure 8.7.5⛶ fullscreen applies the CDF to synthetic and real images, shows the histograms before and after, and follows one gray level through the curve.

A narrow cluster in the input histogram makes the CDF rise steeply there. Applying that CDF as a tone curve spreads the cluster over a wider output range, increasing distinctions among those pixels. If the cluster was mostly noise in a smooth wall, the same operation increases the noise contrast. An image-wide histogram cannot tell the difference between useful texture and unwanted variation. This explains why a mechanically well-defined equalization can give an unattractive photograph, and why limiting the curve's slope or using another rendering criterion can help.

White balance on film🔗⧉

Before sensors, white balance was chosen when the film was bought. A color film's three emulsion layers are balanced so that one particular illuminant records as neutral. Daylight film is balanced for about 5500 K, the mix of sun and sky at midday; tungsten film, sold as Type B, for about 3200 K, the color of studio lamps (Figure 8.10.3⛶ fullscreen). Shoot daylight film under household or studio lamps and everything comes out orange, which is the "before" panel of Figure 8.10.2⛶ fullscreen; shoot tungsten film outdoors and everything comes out blue.

A photographer caught with the wrong film under the wrong light used a conversion filter on the lens. A blue 80A filter lets daylight film record correctly under 3200 K lamps, at the cost of about two stops of light, because it works by absorbing much of the red. An amber 85B lets tungsten film be used in daylight, at the cost of about two thirds of a stop. Photographers reasoned about these filters in mireds, the reciprocal of color temperature scaled by a million,

because a given filter shifts a source by roughly the same number of mireds whatever its temperature. Going from 3200 K (about 312 mireds) to 5500 K (about 182) is a shift of about −131 mireds, which is the 80A's rating, and the 85B is the reverse. Fluorescent tubes, whose spectra are spiky rather than smooth, called for their own filters and often a magenta color-compensating filter to cancel their green cast.

The optical filter is the film-era version of the per-channel gains above, with two differences. It acts on the spectrum before the film records it, so it is a real change to the light rather than a rescaling of recorded numbers, and it costs exposure, which a digital gain does not (though a gain amplifies noise in the channel it raises). And it applies to the whole frame, so a scene lit by two sources, a lamp-lit room with daylight at the window, could not be corrected by any filter on the lens. Film and cinema crews instead filtered the lights themselves, with orange gels on windows or blue gels on lamps, which is the problem Chapter 24.2, Mixed-lighting white balance solves in software.

The two kinds of film also differed in how forgiving they were. A slide is the final image, so a color cast on reversal film is there for good. A color negative is printed, and the printing lab set color filtration for each negative, often with an automatic analyzer, which removed much of a cast after the fact. That printing step was an automatic white balance, run on the negative rather than on sensor data. A digital camera does the same thing per frame, and a raw file defers it entirely, so the choice made at the film counter is now a slider in the editor (Chapter 5.5, Developing and editing a photograph).

Recap 2: non-destructive editing optional🔗⧉

Non-destructive editing stores changes separately from the original samples. It works with raw or rendered inputs, although a rendered input has already committed to earlier processing decisions.

The stored recipe may live in a catalog, a sidecar, or metadata alongside the image. It describes operations, parameters, and masks. The editor recomputes affected stages as needed and caches intermediate results and previews. The original pixel data can remain unchanged even when metadata in the same file changes.

The same raw can then yield a web JPEG and a print by rerunning the pipeline at each output's size, sharpening and color space, and a decision made in the first month can be revised in the sixth. The price is recomputation every time the image is shown, which is why these editors cache previews at several sizes and rebuild them when the recipe changes (Chapter 42.3, Lightroom-style raw developers). The pattern reaches beyond photography: a video editor's edit decision list records cuts and effects over untouched source footage and renders them on demand, and any workflow that would rather replay a derivation than store its output makes the same trade.

Because the recipe is independent of the raw, the same file can become entirely different finished pictures: you simply feed the pipeline a different parameter set, and nothing about the original is consumed in the process (Figure 8.12.3⛶ fullscreen).

Changing parameters avoids repeated lossy saves of intermediate images. Reproduction also requires the original, profiles, external masks, model weights, and the processing version. A future software release may interpret the same sliders differently. Store those dependencies, and keep an exported reference when exact appearance matters.

The demo below lets you run the develop engine directly. It is a working miniature of exactly this pipeline (every stage of the chapter, wired to a Lightroom-style panel) with the edit recipe printed live beside it (Figure 8.12.4⛶ fullscreen).

A non-destructive editor exposes pipeline decisions as an editable recipe. Its screen preview may use reduced resolution or approximations; inspect a full-resolution export for scale-dependent denoising, sharpening, masks, and color conversion. Chapter 8.13, A raw developer in the browser provides a practical setting for those experiments.

The benefits follow directly. The parameter edit is reversible: drag exposure back to zero and it is as though you never touched it, because the original was never altered. It is re-editable: you can revisit a three-month-old adjustment and change a single slider with no accumulated loss from repeated re-saving. The recipe can drive a thumbnail preview or a full-resolution export. Spatial radii and resolution-dependent stages must be scaled appropriately, and the preview can be an approximation to the final rendering. The one thing it demands is speed: recomputing the pipeline every time you nudge a slider only feels interactive if the pipeline is fast, which is why these tools lean hard on the graphics processing unit (GPU) and on carefully optimized image code, a concern we return to in the performance part.

SSIM🔗⧉

Structural similarity (SSIM) Wang et al. 2004 compares local brightness, contrast, and correlation. For windows centered at the same location in the two images, a common form is

The means $\mu_I,\mu_J$ describe local brightness; the variances $\sigma_I^2,\sigma_J^2$ describe contrast; the covariance $\sigma_{IJ}$ describes agreement after subtracting the means. The positive constants $C_1,C_2$ stabilize nearly flat or dark windows. Identical windows score one. Scores can be negative, particularly when local patterns are anticorrelated. Wang et al.'s formulation specifies the window and constants needed to reproduce a score.

SSIM is a map before it is a number. Because the window slides across the image, what SSIM produces first is a per-pixel structural-similarity image. Its displayed values indicate local agreement under the chosen visualization. The single "SSIM" value people quote is just that map's average. The map retains spatial information, because it localizes the damage in a way a scalar never can.

SSIM adds useful spatial information but remains sensitive to window size, scale, and alignment. A luminance-only implementation misses purely chromatic errors. Multi-scale SSIM compares several resolutions. Record the implementation, input range, channel handling, window, and border convention; the name alone is not a complete evaluation protocol.

Two interpolations: domain (shape) and range (color)🔗⧉

The cure has two knobs. Name them precisely, because the method is their product.

Range interpolation (color) is the cross-dissolve we already have: at a location $x$, blend the two colors,

a straight linear blend of pixel values.

Domain interpolation (location) treats a feature point as a vector and interpolates where it is. For a corresponding pair, a point $P$ in $I_0$ and the matching point $Q$ in $I_1$, the in-between location is

At $t=0$ the feature sits where $I_0$ put it; at $t=1$ where $I_1$ put it; in between it travels straight along the segment $PQ$ (Figure 11.3.2⛶ fullscreen). This is interpolation of the domain, the coordinate grid itself, as opposed to the color living on it.

To realize a domain interpolation as an image operation we warp: move pixels spatially while leaving their colors alone, $C'(x,y) = C\big(f(x,y)\big)$. In practice we use the inverse / backward map (for each output pixel, look up where it came from, $\text{out}(x,y)=\text{im}\big(f^{-1}(x,y)\big)$), so every output pixel is filled (no holes) and resampling is well-defined. This is exactly the engine of Chapter 11.1, Warping; morphing is one of its biggest customers, calling it twice per frame (once for each source image).

The morph in one line: interpolate the domain like a vector, then interpolate the range (color). A morph is the product of a shape interpolation and a color interpolation: two independent linear blends, applied together (Figure 11.3.2⛶ fullscreen).

Dynamic range and multiple-exposure capture🔗⧉

Dynamic range is the ratio between the largest and smallest signal a sensor records. At the bright end, charge capacity or the readout chain limits the recorded signal. At the dark end, noise sets a gradual loss of precision rather than a hard black threshold. The usable dynamic range depends on the minimum signal-to-noise ratio (SNR) accepted for the task (Chapter 2.8, Noise, signal-to-noise ratio and dynamic range). That chapter develops both ends: full-well capacity at the top (the maximum number of electrons a pixel can hold), read noise and photon noise at the bottom. The engineering definition places the floor where the signal equals the noise (0 dB), but that is too low for photographic purposes, where 30 dB is considered close to perfect and even 15 dB is severely noisy. Figure 17.1.1⛶ fullscreen shows examples. Like depth of field, quantitative dynamic range is a question of tolerance, to noise in this case. Recall that bright scene values that saturate the photoreceptor may still be read below the maximum because of electronic noise. The ability to capture high dynamic range can further be limited by the veiling glare inside the lens, which lifts the darkest tones before the sensor sees them.

When the dynamic range of a scene exceeds that of the sensor, we capture multiple exposures to span the range. We may need to register these images in space (similar to denoising). Chapter 13.2, Denoising by averaging aligns and averages a burst of equal exposures, Part 12, Matching pixels across space and time covers the registration itself, and Chapter 17.4, Ghosts and motion deals with what moves between the frames of a bracket. We also need to calibrate the pixel values photometrically to bring them into the same absolute linear range proportional to scene-radiance values $L(x,y)$, which may require the calibration of the film or digital camera response curve (the beauty curve in particular, Why scene-linear looks dull in Part 8, Basic image processing and ISP; Chapter 17.2, Camera response and radiance calibration recovers it). Motion, clipping in every frame, or inadequate shadow exposure can leave parts of that estimate uncertain.

Why log space helps🔗⧉

Log luminance makes relative contrast convenient to manipulate. Scaling a log difference by $c$ changes the corresponding luminance ratio $R$ to $R^c$. For $0<c<1$, this compresses the ratio. Multiplication in linear light is already simple; the benefit here is that ratios become differences that can be separated into base and detail.

This is why the local methods of the following chapters work on log luminance: a base and a detail layer of the log image are a large-scale ratio and a local ratio, and compressing the base compresses large ratios while leaving local ones alone (Chapter 18.3, Two-scale and bilateral tone mapping).

Highlight / specular recovery🔗⧉

A last, less obvious instance closes the chapter, because it is inpainting in disguise. A blown highlight (a glaring specular spot on metal, a sky clipped to pure white, a sunlit window) is a region where the sensor saturated: one or more channels reached their ceiling, so the value above that threshold was not recorded. Saturation still supplies a lower-bound constraint, so a clipped region is not quite an unconstrained hole, and highlight recovery is inpainting it: filling the saturated pixels with a plausible surface color using the surroundings and whatever was not clipped as the prior (Figure 21.1.14⛶ fullscreen).

There are two complementary angles. The first is specular–diffuse separation: a highlight is the specular reflection added on top of the underlying diffuse surface, so recovering the diffuse layer beneath it is the goal (cross-ref the illumination-in-a-single-image material). The second, and more directly inpainting-flavored, is per-channel clipping recovery: highlights rarely clip all three color channels at once, so when only the red channel has saturated, the unclipped green and blue still carry two channel measurements and local structure, but do not determine the missing red or full hue, and the missing red can be extrapolated from them and from the surround, a more constrained but still ambiguous inpainting whose prior is "the channels move together."

It lives here for the same L7.8show reason as everything else: recover what the measurement destroyed, destroyed by clipping rather than by masking, but unmeasured all the same. And it has a grounded, non-inpainting cousin: rather than guess the highlight, capture it with a shorter exposure and merge, the high-dynamic-range (HDR) move of trading a guess for a measurement (forward-ref Multiple exposure imaging). Inpainting the clip is what you do when you only have the one frame.

Mixed-lighting white balance🔗⧉

Photograph a room at dusk with a lamp on and the curtains open, and the camera cannot solve the problem with a single global adjustment. The lamp throws a warm, orange light on one side of a face; the window lets in cool, bluish daylight on the other. White-balance for the lamp and the daylit cheek turns blue; white-balance for the window and the lamp-lit cheek turns orange. There is no global gain that neutralizes both, because the right correction is different in different parts of the frame, and an ordinary white balance has exactly one correction to give.

This is where the global white balance of Chapter 8.10, Auto-exposure and auto white balance fails. There, the camera's job was to estimate one illuminant and divide it out with a von Kries diagonal gain $k_R, k_G, k_B$: a single multiply in linear light. That model assumes one light color over the whole scene. Mixed lighting violates the assumption at the root: the illuminant is no longer one color but a field of colors that varies across the image. So we cannot estimate one gain; we have to estimate, at every pixel, which light is doing the lighting there, and correct accordingly.

Correcting each region for its own light🔗⧉

With the mixing map $\hat\alpha(x)$ in hand, the correction is mechanical: at every pixel reconstruct the local light $L(x) = \hat\alpha(x)\,L^{(1)} + (1 - \hat\alpha(x))\,L^{(2)}$ and divide the pixel by it, a per-pixel von Kries gain instead of one global gain. Where the lamp dominates we divide out the warm light; where the window dominates we divide out the cool light; in between we divide out the blend. With accurate estimates and a suitable diagonal model, neutral surfaces in both regions lose their casts at once, the thing a single gain could never do. The art and craft connection is direct: this is what a careful colorist does by hand with local color correction, masking the warm and cool regions and grading each separately. The method does it automatically and continuously, the per-pixel analog of dodging-and-burning the color of the light.

The estimate also supports lighting edits. Replacing $L(x)$ changes the local light color. To simulate switching off a source, however, retain its unnormalized contribution: write $C=\rho(i_1L^{(1)}+i_2L^{(2)})$ and set the appropriate amplitude $i_1$ or $i_2$ to zero. Changing only the normalized mixture does not reproduce the brightness loss. Neither operation predicts new shadow geometry or view-dependent reflections.

Estimate a spatial illuminant map before applying local white balance. Check neutral surfaces under both lights and inspect transitions at shadows. Material assumptions, endpoint errors, and the diagonal color model limit the correction; smoothness by itself cannot identify the light.

What came after🔗⧉

Hsu et al. used known or estimated endpoint colors and a material-color model. Later work pushed on every part of that: removing the user, allowing more than two lights, learning the spatially-varying correction, and building the datasets to train and measure it.

• User-guided lighting and white balance. Boyadzhiev, Bala, Paris and Durand (2012) generalize the idea into an interactive system for user-guided white balance and lighting edits in a single image: the photographer marks intent ("this should be neutral," "this region is lit by that light") and the system propagates a spatially-varying correction with the same smoothness/affinity machinery. The mixing estimate becomes a controllable editing tool, not just an automatic fix.

• Automatic multi-illuminant estimation. Gijsenij, Gevers and van de Weijer (2012) tackle the fully automatic multi-illuminant problem (estimating a spatially-varying illuminant map without user-seeded light colors) by sampling local illuminant estimates across the image and combining them. The hard part they confront head-on is the one Hsu sidestepped: getting the light colors themselves out of the image, not just the mixing between known endpoints.

• Ground truth for mixed light. Beigpour et al. (2014) build a multi-illuminant ground-truth dataset (real scenes captured with per-pixel known illumination) so that spatially-varying methods can finally be measured rather than only shown. As everywhere in this part, a sound evaluation needs data that records the true answer at every pixel, which mixed light makes expensive to obtain.

• White-balance editing and the learned fix. Afifi and Brown (2019) study "when color constancy goes wrong" (what a globally mis-white-balanced image looks like and how to repair it) and their follow-up (2020) learns to edit white balance in camera-rendered sRGB images. That is a related but different problem: correcting a rendered global setting does not itself estimate a per-pixel mixed-light map. Learned spatial corrections must also address the mixture (see Chapter 7.3, Deep learning).

• A large-scale dataset. Kim et al. (2021) release LSMI (Large-Scale Multi-Illuminant), with 7,486 images from more than 2,700 scenes captured using three cameras, the training and benchmarking substrate that the learned spatially-varying white-balance methods had been missing. With LSMI, the per-pixel illumination map becomes a supervised learning target.

The pattern: Hsu's estimate the mixture, then correct each region is the conceptual core, and everything after either removes a hand-set assumption (the seeded light colors, the two-light limit), replaces the smoothness prior with a learned one, or builds the data to do so. It also connects outward to intrinsic-image / illumination decomposition (separating a photo into reflectance and shading/illumination) of which spatially-varying white balance is the color-of-the-light special case.

Measuring and improving a kernel🔗⧉

Start from a release build, a representative input, and a correctness check. Profile to find expensive stages, then test a specific explanation: too much traffic, poor reuse, limited vectorization, a serial reduction, or too little work per task. Change one factor and measure again.

Benchmark both the kernel and the complete path that matters to the application. For steady-state timing, warm up JIT compilation and allocate reusable buffers beforehand; report cold-start compilation and allocation separately when they affect the user. Synchronize asynchronous work before stopping the timer. Run enough repetitions for stable measurements, consume outputs, and report variation as well as a central value. Record hardware, compiler, target flags, dimensions, precision, thread count, and whether inputs were cache-resident. Sustained camera workloads also need thermal steady-state measurements.

A profile becomes useful when it identifies a mechanism. False sharing, for example, means workers update different values on one cache line, causing coherence traffic between cores even though the program shares no individual value. Padding or assigning contiguous disjoint regions can change that traffic. Load imbalance is different: one worker receives more work and holds up completion. More threads do not repair either problem automatically. They need changes to data layout or task assignment.

Hand-off: from rays to radiance fields and generation🔗⧉

This chapter is the seam. Behind it lies the classical light field (capture, index, refocus) and the lumigraph's lesson that a prior buys back samples. Ahead lie two larger stories that grow straight out of it. The radiance-field idea, sketched here only as NeRF-the-context, becomes its own subject in Chapter 36.7, Photos → radiance fields and Gaussian splatting (NeRF, 3DGS): how learned volumetric scenes are built, accelerated (the many-evaluations cost is the thing everyone attacks), and turned into the workhorse of 3-D reconstruction. And the move from fitting a scene to generating one means sampling plausible scenes and views rather than reproducing measured ones, and it is the subject of Chapter 7.4, Generative AI and diffusion and the 3-D part, where the learned prior stops merely interpolating captured rays and starts inventing them. Both inherit this chapter's frame: a camera takes a scene to an image by a projection (a forward image-formation model, linear in ray radiance for fixed optics, though NeRF rendering is nonlinear in its unknown density), and a learned scene representation is precisely a function trained to invert that projection, to answer "what scene would have produced these views?" so that new views can be rendered from it. The light field network is the cleanest statement of that frame on the light-field side: the network is the scene, the ray is the query, and a photograph is one forward pass away.

Recap: big lessons in this lecture🔗

💡 Big lesson (L2.6) — the multiplicative world🔗

Much of imaging is multiplicative, not additive. The light reaching the eye is illumination times reflectance; contrast is a ratio of intensities; a shadow darkens a region by a factor, not by subtracting a constant. Because the structure is multiplicative, the natural tools are multiply, divide, and log, not just add. This is why, again and again, we will work with ratios and with logarithmic encodings rather than raw linear values. It is also why white balance is hard: to recover an object's true color we must divide out the unknown illuminant $S(\lambda)$ from the product $C(\lambda)$, undoing a multiplication we never observed directly. (The additive counterpoint, light from different sources adding, and what it means for choosing an encoding, comes in Color technology.)

💡 Big lesson (L2.26) — a pixel value is an integral🔗

No pixel measures "the" value at a point in space, time, or color: it reports an integral over everything the photosite cannot tell apart. Four axes appear explicitly above: the spatial footprint (the pixel's area, plus the optical blur smeared across it), the exposure time (a window, not an instant), the spectrum (wavelength, weighted by the channel's spectral sensitivity), and the solid angle of rays the aperture admits; a fifth, polarization, is usually lumped in too. Naming all the axes at once is what makes motion blur, defocus, color, vignetting, and aliasing the same phenomenon (a finite integral standing in for an idealized point sample), and the width of the integral along each axis is a knob the rest of the book spends its time controlling.

💡 Big lesson (L2.35) — sensors convert photons to electrons linearly🔗

A photosite is at heart a photon counter: each absorbed photon frees a charge carrier, an electron or equivalently a hole (a missing electron), and the well accumulates them in direct proportion to the light that arrives. So the raw signal is linear in scene radiance, before any gamma or tone curve. This linearity is not a detail; it is what makes raw arithmetic trustworthy: HDR merging adds exposures, white balance is a per-channel multiply, flat-field correction is a divide, and all three are valid only because the numbers are proportional to light. Display encoding (gamma) deliberately hides this linearity for the eye, so computation should undo it and work in linear light.

💡 Big lesson (L3.12) — the eye discounts the color of the illuminant🔗

Chromatic adaptation and scene interpretation help stabilize surface color across illuminants. Per-channel cone gains are a useful approximation. Camera white balance makes a related rendering choice for the output conditions, with accuracy limited by the measurements and estimated illuminant.

💡 Big lesson (L3.13) — capture and viewing conditions both affect color appearance🔗

A photograph is viewed under conditions different from those at capture. Its appearance depends on the display or print, the surround, adaptation, and cues within the picture. White balance and rendering should target those conditions and the photographer's intent; viewers do not automatically undo the captured illuminant.

💡 Big lesson (L3.7) — the eye's dynamic range is enormous over time and modest at any instant🔗

Distinguish sensitivity across adaptation states from the range visible in one scene under specified conditions. Local adaptation, glare, spatial structure, and the task all affect that range. A single stop count is insufficient to compare an eye, camera, and display.

💡 Big lesson (L3.10) — contrast depends on a reference level🔗

Detectable increments often scale approximately with the background in the photopic range. State which contrast definition, stimulus, and viewing conditions a measurement uses. Absolute pixel error and perceived difference answer different questions; neither replaces the other.

💡 Big lesson (L3.11) — surface appearance can remain stable as illumination changes🔗

Surface-lightness and color judgments can remain relatively stable as illumination changes. This constancy is partial and uses several cues. It differs from recovering the complete reflectance spectrum, and it does not imply that brightness or the appearance of lighting stays fixed.

💡 Big lesson (L4.8) — there is no single RGB: gamut versus bit budget🔗

Choose a working gamut large enough for the content and operations, with sufficient numeric precision. Choose a delivery space the receiving system can interpret. Wider primaries alone do not improve an image, and converting an sRGB image to a wider space does not recover colors previously clipped.

💡 Big lesson (L5.1) — photographers measure light in stops🔗

A stop is a factor of two. Time steps by two and f-number by $\sqrt{2}$. ISO and a software exposure slider can change rendered brightness by a stop without collecting another photon.

💡 Big lesson (L7.5) — factorization is inverting a product, and it is fundamentally ill-posed🔗

Factorization problems seek to decompose a number, or an image, into a product of two unknown pieces (scalars, or whole matrices) given only the product. This is fundamentally ill-posed: many factor pairs multiply to the same result, and unconstrained formulations often admit the degenerate "put everything in one factor, leave the other the identity" split. Additional constraints, calibration, or priors breaks the tie. In the fully general case it is our linear problem $\mathbf{y}=A\mathbf{x}+\mathbf{n}$, except we must now solve for the operator $A$ as well as the unknown $\mathbf{x}$. Examples include white balance, intrinsic images, matting, and blind deblurring.

💡 Big lesson ((L8.1) · Dither: trade structured error for unstructured noise)🔗

Dither replaces visible quantization structure with noise. Independent stochastic rounding produces approximately white noise; error diffusion shapes the error spatially and can leave patterns. Dither preserves an average under stated assumptions, not every pixel or every local region.

💡 Big lesson (L8.2) — loop in memory order (innermost index varies fastest)🔗

Prefer traversal that reuses nearby stored values. For scalar loops over packed HWC, use y, x, c order; for packed CHW, use c, y, x. Actual strides, tiling, and vectorized kernels can change the best schedule. Measure before changing the representation.

💡 Big lesson (L4.1) — pixel numbers are meaningless on their own🔗

The numbers in an image array carry no meaning by themselves. The same triple (0.5, 0.2, 0.1) is a different color, and a different amount of light, depending on two things the array does not contain:

• its color space: which primaries and white point those R, G, B refer to (sRGB? Adobe RGB? Display P3? a camera's native raw space?); and
• its value encoding: how a stored number maps to light (linear, gamma, log, or a camera "look" curve).

Strip those two away and you are left with a grid of anonymous numbers. Every operation that matters later (white balance, HDR merging, blur and deblur, any color conversion) silently assumes a particular answer to both questions, and gets the wrong result if the assumption is wrong. Hence the discipline of the whole book: never touch a pixel without knowing its color space and its value encoding. These are two common causes of otherwise puzzling image differences.

💡 Big lesson (recurrence of (L4.6) and (L4.4)) — mind the encoding

Use the domain expected by the operation. Decode the transfer function before linear-light arithmetic, and encode appropriately for display. Decoding a rendered file does not undo tone mapping or restore scene-linear measurements.

💡 Big lesson (L4.6) — additive vs. multiplicative → choice of encoding🔗

Exposure multiplies linear-light values; independent contributions of light add. Log coordinates turn exposure ratios into differences. An offset in encoded values is a distinct appearance control. Choose the domain from the operation and its model.

💡 Big lesson (L8.3) — exposure scales linear light (a multiply) or shifts log values (an add)🔗

Exposure scaling multiplies linear-light values by $k=2^s$ or adds $s$ in base-2 log units. For a pure gamma encoding $V=L^{1/\gamma}$ it multiplies V by $k^{1/\gamma}$. For an arbitrary transfer function, decode, scale, and encode. Output clipping is a separate operation.

💡 Big lesson (L8.4) — contrast is a pivot: choose what stays put🔗

A contrast change rotates values about a chosen pivot: $L_\text{out} = p + g\,(L_\text{in}-p)$. The pivot $p$ is exactly what is left unchanged, and it also decides whether the picture gets brighter or darker overall: a high pivot pulls most of the image down (darkening), a low pivot pushes it up (brightening). Unlike exposure, contrast is not radiometrically meaningful, it is a stylistic stretch, so the encoding matters less (no single space is "correct"); what matters is a sensible pivot, and the sensible pivot does depend on the space: roughly 18% mid-gray in linear light, the numeric midpoint in gamma or log. Pick the pivot deliberately: it is the whole character of the operation.

💡 Big lesson (L8.6) — store the recipe, not the result🔗

Preserve original samples and store the operations and parameters separately. This makes adjustments revisable without repeatedly quantizing intermediate images. Also preserve the processing version and dependencies; a recipe is reproducible only while its ingredients and interpretation remain available.

💡 Big lesson (L2.46)🔗

A common sensor dynamic-range measure divides the saturation signal by the read-noise standard deviation, in consistent units. A stricter quality threshold gives a smaller usable range. Bracketing extends coverage when its exposures provide reliable, registered measurements of the same scene.

💡 Big lesson (L4.6) — additive vs. multiplicative → work in log🔗

Choose an encoding for the operation. Add independent light contributions in linear light. For relative-contrast decomposition, log luminance turns ratios into differences, so a base and a residual can be combined by addition.

💡 Big lesson (L18.2): tone-map in the log domain when the operation acts on ratios🔗

Log luminance is useful when the operation should act on intensity ratios, as in additive base/detail decomposition or gradient attenuation. Other operators, including Reinhard's $L/(1+L)$, can be formulated directly in linear luminance. Match the representation to the operation; do not take logarithms of nonpositive values.

💡 Big lesson (L29.4) — separate the algorithm from its execution schedule🔗

The algorithm specifies the result; the schedule controls loop organization, parallelism, storage, and recomputation. Changing the schedule can change both traffic and executed arithmetic, especially at tile halos. Halide makes these choices explicit so you can compare them without obscuring the algorithm.