4.9 Color TLDR⧉
Human color vision reduces any light to three numbers, so color is a three-dimensional space, and every "color space" you will meet, LMS, XYZ, the RGBs, YCbCr, Lab, HSV, is a different set of coordinates on that same 3-D space. Almost everything that follows is choosing coordinates and converting between them with small matrices, plus a few nonlinear curves layered on for storage and for perception. Color science was built on the task of reproducing color more than measuring it, so its perspective is more heavy on color synthesis and the choice of primary colors than a pure measurement approach would warrant, which can create confusing for photography.
4.9.1 Physical color⧉
Before any eye or sensor, color is physical: a light is a spectrum, its power at each wavelength (the spectral power distribution). For a surface, the light that reaches the eye is not the surface's own but the illumination times what the surface reflects, $C(\lambda) = S(\lambda)\,\rho(\lambda)$: the illuminant spectrum $S(\lambda)$ multiplied wavelength by wavelength by the surface reflectance $\rho(\lambda)$. An apple looks red because its reflectance is high in the reds and low elsewhere, but only when there is red light to reflect: under a pure green lamp the same apple is nearly black. Self-luminous things (a screen, a flame, the sky) skip the product and emit their spectrum directly. Everything after this reduces that spectrum to a few numbers. Details: Chapter 4.1, Linear color reproduction and CIE XYZ.
4.9.2 Cones, LMS, and metamerism⧉
The eye has three types of color-sensitive photoreceptors. Given a physical spectrum, their response $L, M, S$ is a projection (dot product) of the light onto three sensitivity curves. That is, it's the integral of the light spectrum times the sensitivity of each cone. Because the sensitivity curves overlap, the three axes, $L, M, S$, are non-orthogonal (there is cross-talk), and because light cannot be negative, not every point is reachable. And because only three numbers survive, many different spectra collapse to the same triple: metamers. That is, metameters are different physical spectra that project on the same $L, M, S$, and are therefore perceived as the same color. This is exactly what lets a three-primary screen reproduce any color. Details: Chapter 4.1, Color space 101.
4.9.3 Light and chromatic adaptation⧉
What the visual system is really after is the surface, its reflectance and tone, not the light that happens to fall on it. So it discounts the illumination, and does so by dividing it out on two axes. It adapts to the ambient level, rescaling its sensitivity to the prevailing intensity so a scene reads much the same in dim and bright light (light adaptation). And it adapts to the color of the light, scaling the three cone gains toward the illuminant so a white page looks white under warm tungsten and cool daylight alike (chromatic adaptation, the von Kries mechanism). Both corrections are multiplicative, a gain per channel, which is why human vision is largely invariant to multiplication (L3.8): scale the light up or down, or tint it, and the perceived color and tone of surfaces barely move. Illumination is not the only thing the visual system discounts, though: perceived color and tone also depend on the surround, the neighboring tones and colors, so one gray looks lighter against black than against white and a patch shifts toward the complement of its background (simultaneous contrast). Constancy is an inference from spatial context, not a readout of absolute values. This is the perceptual basis for white balance and for working in ratios and logs. Details: Chapter 3.3, Adaptation and constancy.
4.9.4 XYZ⧉
In an ideal world, color science would be built on $L, M, S$, but for historical reasons (it took a while to have technology to measure cone responses), the standard we all build on is CIE XYZ. It is a linear transform of LMS, one $3\times3$ matrix away; its coordinates are semi-arbitrary, with one exception, $Y$ is defined to be luminance. It was originally built through color reproduction experiments rather than by measuring cones, but you do not need that history to use it: treat XYZ as the standard, device-independent coordinate system everyone converts through. Details: Chapter 4.1, Color space 101.
4.9.5 Chromaticity and gamut⧉
For visualization purposes, it is convenient to drop brightness and use a 2-D chromaticity diagram. The standard version starts with XYZ and divided by the sum of the three numbers. Physically realizable colors (spectra with no negative light) occupy a horseshoe shape with monochromatic light at the boundaty. A device's reproducible colors, the mix of its primaries, form a convex region (a triangle for three primaries) inside it: its gamut. Colors outside the triangle cannot be made and must be mapped in, which is called gamut mapping. Details: Chapter 4.1, Color space 101, Chapter 4.7, Synthesizing color.
4.9.6 Linear RGB, and why there are many⧉
There are many named RGBs (sRGB, Adobe RGB, Display P3, Rec.2020, ProPhoto). Getting to one RGB space from XYZ is again a single $3\times3$ matrix.
People define different RGBs for three practical reasons: different display primaries (so different gamuts), and different trade-offs of bit budget and bandwidth (a wider gamut spreads the same bits thinner). Pick by what you need to cover and how many bits you can spend. Details: Chapter 4.4, RGB color spaces.
4.9.7 White balance and white point⧉
Two different "whites." White balance is the camera's correction for the color of the scene's light: a change to the data. The white point is the neutral the space is normalized to for the display conditions (D65, D50). They are distinct, and color appearance genuinely depends on context, starting with the ambient illumination the eye adapts to, so "the color" is never absolute. In an ideal world, white point would be involved only late in the pipeline but because color science is so focused on repdoduction, it is involved earlier than this author would like. Details: Chapter 4.2, White balance and white point.
4.9.8 Gamma is for encoding (and doesn't hurt processing much)⧉
Values are usually not stored linearly. They are gamma-encoded, a power curve that puts quantization levels where the eye can use them so eight bits look smooth. Its main job is storage, but it also happens to help some processing, or at least not hurt, by making steps roughly perceptual (recall that human vision is largely invariant to multiplication, L3.8). This is a big, subtle topic (it is not the same as log, and "gamma correction" is a misnomer) that the book returns to repeatedly. Details: Chapter 4.3, Encoding: linear, gamma or log.
4.9.9 sRGB⧉
The one color space you meet most is sRGB: the standard behind essentially every JPEG, PNG, and web image, and the thing to assume whenever a file arrives with no profile. It fixes the Rec. 709 primaries and a D65 white and is stored gamma-encoded (the piecewise curve near $2.2$ of the previous section), so its numbers are perceptual code, not linear light. Linear sRGB is a single fixed $3\times3$ from XYZ, and therefore one more $3\times3$ from the LMS cones, which is why "an sRGB value" doubles as shorthand for "an ordinary color"; the actual matrices are in Color space 101#Linear RGB. Decode the gamma before any light math. Details: Chapter 4.3, Encoding: linear, gamma or log.
4.9.10 RAW files⧉
A camera RAW file is the opposite end: scene-linear sensor readings, straight off the photosites before any tone curve, which makes it the right place to start physical work (HDR, denoise, white balance). But raw camera RGB is usually not a clean linear transform of LMS or XYZ, because the sensor's color filters only approximate the cones, overlapping differently, so no fixed $3\times3$ maps them exactly onto human color. The developer fits a best-effort color-correction matrix per camera model to get close, and that fitted, imperfect map (not an exact identity) is why two cameras render the same scene a little differently. Details: Chapter 4.6, Sensing color: multiplexing strategies, Chapter 45.14, DNG, the Digital Negative.
4.9.11 Log is for high dynamic range, video, and some processing⧉
A third encoding, log, stores the logarithm of the light, so equal steps are equal ratios (stops) rather than equal amounts. That makes it the natural home for anything multiplicative: it turns a change of light into a simple shift (the multiplication-invariance the eye already has, L3.8), which is why it is a good working space for HDR tone mapping, why high-end and prosumer video cameras shoot in log profiles (S-Log, Log-C, V-Log, and the ACES pipeline), and why perceptual, illumination-invariant constructions like the Chong color space are built on it. Gamma is really just log's better-behaved cousin near black. The one hazard lives exactly there: $\log 0 = -\infty$, so guard against zero and negative values (add a small floor) before taking a log. Details: Chapter 4.3, Encoding: linear, gamma or log, Chapter 8.11, File formats and compression.
4.9.12 Opponent (luma-chroma) spaces⧉
Human vision recodes the cones into one luminance channel and two opponent color channels (red vs green and blue vs yellow), and that is again just a linear transform of linear LMS (so it could be computed from linear RGB). But the versions used in practice, YCbCr and related spaces, apply their matrix after gamma, on encoded values, for convenience and history. That is a little sketchy (the result is luma, not luminance) but perfectly fine for its purpose, storage and compression. However be careful if you want to do more serious processing. YCbCr is central to JPEG compression because human spatial perception is different along brightness vs color opponents. Details: Chapter 4.5, Advanced color, Chapter 8.11, File formats and compression.
4.9.13 HSV is for UI⧉
HSV / HSL re-parameterize RGB into hue, saturation, and value/lightness with simple arithmetic on the encoded values. They are wonderful for picking colors in an interface and misleading for anything quantitative. Use them for sliders, not for math or perception. When you want those same hue, saturation, and lightness controls but behaving perceptually, reach for the cylindrical form of a uniform space instead: OKLCH (the cylindrical form of Oklab, and now a CSS color function) is the best modern default, with CIELCh (cylindrical CIELAB) and HSLuv as alternatives. All give hue, chroma, and a real lightness, so equal slider steps look equally different and changing lightness does not lurch the saturation the way HSV's "value" does. Details: Chapter 4.5, Advanced color.
4.9.14 Perceptually uniform spaces are for differences⧉
CIELAB and Oklab deliberately warp color so that equal distances look equally different. That makes them the right tool when you need to measure a color difference, set a tolerance, draw an even gradient, or map a gamut. Use them when the question is "how different do these look?" A logarithmic relative, the Chong color space, is both near perceptually uniform and illumination-invariant, so a change of light becomes a simple shift, which makes it good for multiplication-invariant processing as well as for measuring differences. Details: Chapter 4.5, Advanced color.
4.9.15 The full chain: from physical color to perceptual color to color names and memory⧉
Illumination × reflectance. Light leaves a surface as the illuminant spectrum times the surface reflectance (the Lambertian approximation; self-luminous sources skip this step and emit directly), giving the stimulus spectrum that reaches the eye.
Trichromatic projection. The three cones integrate that spectrum against their sensitivity curves, collapsing it to $L, M, S$. Metamerism is born here: every spectral detail beyond three numbers is discarded, so different spectra can become the same color.
Chromatic adaptation (von Kries gain). Early, at the photoreceptor level, each cone's gain is scaled to the prevailing light, which largely divides the illuminant's color back out. This is the mechanism behind color constancy and behind a camera's white balance.
Opponent recoding. The retina reorganizes $L, M, S$ into one luminance channel and two chromatic axes, a linear map (the DKL space), which is why luma-chroma codes match the eye and why chroma can be compressed.
Cortical processing. In the visual cortex the signal is retuned: hue is sharpened, color appearance (brightness, colorfulness, contrast effects) is computed, and color constancy is refined using spatial context, the surround and the illuminant inferred across the whole scene.
High-level color. Finally language and memory take over. Color naming sorts the continuum into a handful of basic categories (about eleven in English), and those categories are believed to be organized around prototypes: a single best example of "red" or "green" that observers across very different languages largely agree on, with fuzzy, negotiable boundaries in between (Berlin and Kay; Rosch). Categorical perception follows, two colors straddling a boundary are easier to tell apart than two equally-spaced colors sitting inside one category. Memory colors are the other half: familiar things are remembered as more saturated and more ideal than they actually measure, so the sky is recalled as bluer, grass as greener, and skin as warmer and more even than any photograph of it. That is exactly why preferred color reproduction nudges those particular hues toward the remembered version rather than the measured one (Chapter 4.5, Advanced color).
The engineering half of this part lives in the first three steps; the perceptual chapters own the last three. Details: Chapter 3.2, Perceptual color and trichromatic vision, Chapter 3.3, Adaptation and constancy.
4.9.16 Color and encoding in the real world⧉
Theory says: do physical operations in linear light, store and show in a perceptual encoding, and always know which space your numbers are in. Real code ranges from religiously following that to blithely ignoring it. This section walks the range, because seeing where the discipline is kept and where it is dropped is the fastest way to internalize why it matters.
In camera ISPs⧉
A camera ISP is the most disciplined color pipeline most images ever pass through, and its structure is remarkably consistent across implementations (Recap ISP, non-destructive editing#Real ISPs surveys a dozen open-source ones in detail). The physically-linear stages, black-level subtraction, white balance, demosaic, the color-correction matrix, and shot-noise denoise, are all done in camera-linear RGB, because that is where the numbers are proportional to photons and the operations are meaningful. Only then does a rendering transform (a tone curve, or filmic/sigmoid/AgX/OpenDRT) cross into a display encoding. The one genuine point of variation is where that gamma boundary sits: scene-linear raw editors (darktable, vkdt) hold everything linear until a single final transform; hybrid editors (RawTherapee, ART) deliberately drop into Lab, CAM, and gamma-bounded spaces for photographic controls; and hardware/reference ISPs (Mini-ISP, openISP) apply sRGB gamma right after the color matrix and then run a whole encoded-domain stage of YUV luma/chroma denoise, sharpening, saturation, and contrast. That last pattern is the important corrective: processing in gamma-encoded YCbCr is not sloppiness, it is a deliberate perceptual stage, done after the noise-critical work was finished in linear light.
In PyTorch and OpenCV⧉
General-purpose imaging libraries take the opposite stance: they are mostly agnostic, and the encoding contract is the programmer's problem. A PyTorch tensor or an OpenCV Mat is just an array of numbers; imread on a JPEG hands you decoded, usually sRGB-encoded BGR values, and nothing in the type system or the API reminds you of that. Most operators (a convolution, a resize, a matmul) will run happily on those encoded values and produce something plausible, so the burden of deciding "should this be linear?" falls entirely on you. This is a reasonable library design, not a bug, but it means the correctness of color handling is exactly as good as the caller's discipline.
The exception proves the rule: the specialized pipelines do manage encoding, because they must. OpenCV's HDR module (createMergeDebevec, createTonemap) recovers a camera response and works in linear radiance, precisely because HDR merge is a physical operation. But right beside it, OpenCV's panorama stitching is a cautionary tale: its detailed pipeline applies gain exposure compensation and multiband blending, operations that look radiometric, directly to the decoded sRGB samples it was handed. Blending and exposure-matching in display-encoded coordinates is not colorimetrically correct, and for a pleasing LDR panorama it is usually good enough, but it is a clear case of physically-motivated math run on the wrong numbers because the library never linearized them.
Computational-photography pipelines⧉
Research pipelines split cleanly by whether the math forces the encoding question into the open.
HDR people are careful, because they have to be: merging exposures and tone-mapping are defined on radiance, so serious HDR code linearizes first and works in scene-linear light throughout (the merge chapters make this explicit). Panorama software spans the whole range: OpenCV blends encoded samples (above), while Hugin is the mature counterexample, its photometric optimizer fits a camera response curve (an EMoR model) and estimates exposure, vignetting, and white balance precisely because exposure matching and HDR stitching fail otherwise; and Enfuse deliberately optimizes appearance rather than radiance (exposure fusion after Mertens et al., optionally blending in CIECAM02), a useful reminder that "physically correct" is not the only valid goal.
Deblurring is the strictest case, because the forward model leaves no wiggle room. Optical blur is a linear convolution of scene radiance, observed = kernel latent + noise, and convolution commutes with neither the gamma curve nor clipping: a blurred edge in linear light is not* the gamma-encoding of a blurred edge in gamma space. So classical blind-deconvolution code is unusually explicit about it, the Fergus/Hu implementation removes gamma before estimating anything (its GAMMA_CORRECTION defaults to $2.2$, set to $1.0$ for RAW, with an optional measured response map and a saturation mask), because the gradient priors and the kernel are defined on linear intensities. Modern learned deblurring (NAFNet, MPRNet, DeblurGANv2) quietly sidesteps the physics: it trains on stored RGB benchmark pairs (GoPro, RealBlur), so the encoding is whatever the dataset baked in, and "gamma" is not a stated assumption but an implicit property of the training distribution, which works well until the test images come from a different encoding than the training ones.
Computer graphics and lighting simulation⧉
Rendering is where the linear-light discipline is taken most seriously, because a lighting simulation that adds and multiplies radiance is physically wrong the instant the values are not linear. The field's own literature is emphatic about it: textures that represent reflectance (albedo, diffuse maps) must be decoded to linear on load, all shading and light transport run in linear radiance, and the final image is gamma-encoded only at output. This is the standard "linear workflow," laid out in Physically Based Rendering (Pharr, Jakob & Humphreys, 3rd ed. and 4th ed.), argued directly in GPU Gems 3's "The Importance of Being Linear", codified in Real-Time Rendering (4th ed., Akenine-Möller et al.), and worked out for physically-based lighting in Rendering with Radiance (Ward Larson & Shakespeare). The graphics community internalized decades ago what casual imaging code still gets wrong: color textures are perceptual data to be decoded, light is linear, and the encoding goes back on last.
Gamma, cross-fading, and linear operations⧉
The single most common way the encoding boundary bites is the most innocent-looking: interpolating between two colors. A cross-fade, a gradient, an alpha blend, an antialiased edge, all average two colors, and averaging is a linear-light operation. Do it on gamma-encoded values and you do not just get a slightly-off brightness, you get false intermediate colors: fade encoded red to encoded green and the midpoint goes through a muddy dark brown instead of a bright yellow, because the encoding pulls the sum down before the two channels can both be high at once. The fix is the usual one, decode, interpolate, re-encode, and the difference is not subtle once you look (John Novak's "What every coder should know about gamma" is the definitive visual demonstration).
Flat patches make the argument; photographs make it land, because a photograph has thousands of chromaticities and the error goes where the color is. The next figure runs the same three averages on two pictures, and maps what the choice costs in chrominance alone (the figure).
4.9.17 What color space and encoding to do what?⧉
Faced with a concrete task, which space and encoding should you use? The answer almost always follows from a few questions about what the operation is, and most color bugs come from not asking them. The scenario families:
- Anything to do with physical light (adding sources, simulating a lens, merging exposures): linear light, because light adds and only light-proportional values add correctly.
- Anything where linearity matters (blur, resize, average, alpha compositing, gradients): linear again, for the same reason, an average is only meaningful on light-proportional values.
- Sensor and raw processing (demosaic, white balance, color-correction matrix, raw denoise): camera-linear or raw, with a noise model, because these model the physics of capture.
- Compression and coding (chroma subsampling, block transforms, video): the codec's opponent space (YCbCr, ICtCp) with its declared transfer function, because that is where perceptual bit allocation lives.
- When the range is high (HDR capture, tone mapping, wide-latitude grading): scene-linear or log, because 8-bit gamma runs out of precision and log makes stops uniform.
- When the goal is perceptual (color differences, gradients, gamut mapping, hue edits): a perceptually-uniform space (Oklab, CIELAB), because that is where distance means what you want.
- Display and delivery: the target's encoded space (sRGB, Display P3, PQ, HLG), tagged, because that is what the viewer's stack expects.
- When in doubt: gamma-encoded sRGB is a perfectly reasonable default. It is wrong mainly for the physical-light and high-range cases above, which are exactly the ones where you should already be paying attention.
The table below is the same advice at a glance.
| Task | Good space / encoding | Avoid |
|---|---|---|
| Blur, resize, average | Linear RGB or XYZ | sRGB/display-coded values |
| Alpha compositing | Premultiplied linear RGB | Straight alpha in sRGB |
| Render lighting | Spectra or linear RGB | Gamma-coded RGB |
| Raw denoising | Raw or camera-linear RGB with noise model | Display RGB |
| White balance | Raw gains or camera-linear RGB | Lab, HSV/HSL |
| Demosaicing | Raw mosaic / camera-linear data | Display RGB |
| Color correction matrix | Linear camera RGB | Gamma-coded RGB |
| Gradient / palette interpolation | OKLab or OKLCH | sRGB interpolation |
| Perceptual color differences | OKLab, CIELAB, CAM16-UCS, Jzazbz | Raw RGB distances |
| Hue/chroma editing | OKLCH, CIELCH, Lab-like spaces | Naive RGB channel edits |
| Exposure adjustment | Linear RGB, scene-linear RGB, or log RGB | Display RGB without care |
| Contrast / tone editing | Log RGB, Lab L, luma, or display-referred RGB | Unspecified RGB |
| Sharpening | Linear RGB for physical model; luma for look | Blind sharpening of chroma |
| Deblurring | Linear RGB or linear luminance | Gamma-coded RGB |
| HDR merge | Linear radiance / scene-linear RGB | JPEG code values |
| Tone mapping | Scene-linear HDR input → display output | Untagged RGB |
| Chroma subsampling / compression | YCbCr, ICtCp, codec-defined spaces | Scene-linear RGB |
| Video processing | YCbCr/ICtCp plus declared transfer function | Untagged RGB or unlabeled YUV |
| ML image restoration | Match training data and loss explicitly | Pretending encoding is irrelevant |
| Display / web output | sRGB, Display P3, Rec.709, PQ, or HLG | Linear RGB saved untagged |
| Print output | ICC-managed CMYK/RGB print profile | Untagged display RGB |
The same list, by space⧉
Read the other way, here is what each of the main spaces is for:
- Linear RGB (scene-linear, or a linear working RGB such as linear sRGB / linear Rec.2020): the space for light. Everything physical, blur, resize, compositing, lighting, HDR merge, belongs here. Never store it in 8 bits (it bands), and never show it without encoding.
- sRGB (gamma-encoded): the safe storage and delivery default, the web's lingua franca. Great for 8-bit storage and display; wrong for physical math until you decode it.
- Adobe RGB / Display P3: wider-gamut delivery spaces for capable displays and print prep. Same encoding discipline as sRGB, with more saturated reds and greens.
- ProPhoto RGB: a very wide-gamut editing working space (gentle 1.8 gamma). Use it with 16 bits for raw editing, never for 8-bit delivery.
- YCbCr / YUV: the compression/opponent space, luma plus two chroma for subsampling and codecs. An encoding-domain space, neither perceptual nor linear; ideal for storage, wrong for color math.
- ICtCp: the HDR opponent space (PQ-encoded LMS, then opponent). Constant-luminance and hue-stable across the HDR range where YCbCr drifts.
- CIELAB: the classic perceptually-uniform space, lightness plus opponent chroma, for ΔE differences, gamut mapping, and tolerances.
- Oklab / OKLCH: the modern perceptually-uniform space, better blues than Lab and cheap. The right default for gradients, palette interpolation, and hue/chroma edits.
- CIECAM16-UCS, Jzazbz: color-appearance and HDR-uniform spaces for the most demanding difference and appearance work.
- HSV / HSL: UI spaces for picking colors, never for measurement.
- XYZ: the device-independent hub everyone converts through, rarely a working space, always the common reference.
The whole part reduces to a reflex worth building: before an operation, ask what it is doing to the light, and put the numbers in the space where that operation is simple and correct.