array, basic representation and manipulation

• an image is an H×W×C array: a 2D grid of pixels, each a vector of C channels (usually R, G, B)

• sometimes extra channels, for example hyperspectral or for transparency — more in compositing chapter

• warn that it can be a problem when reading PNG if you assume just RGB and you get more channels

We'll focus on floats in [0,1]

8 bits are used on disk and for fast computation but dealing with 8 bit arithmetic is a pain. We discuss it in later chapter (ref).

• side bar: **how floating point works** — a float stores a number as **sign × mantissa × 2^exponent**, binary scientific notation: the **exponent** slides the binary point so the same ~24 mantissa bits (32-bit `float`) give **relative** precision across a huge range. What this means for images:

• **precision is relative** — fine resolution near 0, coarser for large values; a good match for **linear radiance / HDR** (and roughly for how we perceive light)

• values **outside [0,1] are fine** (super-whites, the negative lobes of a sharpening filter) — nothing clamps until you output

• `0.1` is **not exact** in binary, and adding numbers of very different magnitude **loses the low bits** (a long sum of many small values drifts) — usually negligible for images, occasionally not

• **inf** and **NaN** (`0/0`, `log` of a negative, `0×inf`) **propagate** and silently poison an image — a classic bug source; check for them

• `float32` is the working default; **`float16`/half** halves memory (ML, some HDR pipelines), `float64` is rarely needed for images

• [forward ref: the pains of **8-bit integer** arithmetic — overflow, rounding, no headroom — later chapter]

• **reminder** — 💡 **L2.49, quantization is rarely the real problem** (the lesson is introduced in FUNDAMENTALS → [[Image Formation and linear perspective#Noise|Noise]]): with enough bits and a sane encoding it's **noise and dynamic range** that bite, not the number of levels — which is exactly why **floats with headroom** are the sane default here.

• **how many bits for what** (match the depth to the *job*):

• **8-bit integer** — **display and delivery**, and compact storage. ~256 levels is enough *perceptually* **only because gamma** puts the codes where the eye is sensitive (the darks — see *encoding* below); but it has **no headroom** and **rounds on every operation**, so it's a poor *working* format.

• **10–14-bit** — what **camera raw** delivers, sized to the sensor's dynamic range (and to dual-conversion-gain / HDR sensors, FUNDAMENTALS).

• **16-bit integer** — the safe **intermediate for editing**: repeated operations don't accumulate visible **banding / rounding** the way 8-bit does (Photoshop's "16-bit mode").

• **float32 (linear light)** — the sane **working / HDR** format: relative precision, values allowed **outside [0,1]**, nothing clamps until output. **float16 / half** halves memory for ML and HDR pipelines.

• **the rule of thumb**: **more bits — and float — wherever you do *math* or hold *high dynamic range*; few bits only at the display / output end.** Spend precision where rounding and range actually hurt.

• 💡 **bit-depth and encoding decisions go together.** How many bits you need depends on the **encoding** you store them in, because a big part of **gamma's job is to give you better *uniform* quantization**: it spaces the available codes the way the eye perceives brightness — fine steps in the darks, coarse in the highlights — so **8 *gamma*-encoded bits** look smooth where **8 *linear* bits** would band badly in the shadows. (That is not gamma's *only* job — it also matches display response and historical pipelines, see *encoding* just below — but "make a few bits go further" is a big one.) The converse: you do **math in linear**, and linear light wants **more bits / float** to avoid that banding. So don't pick a bit depth and an encoding separately — **choose them as a pair**.

• 💡 **Big lesson (L2.52 · dither: trade structured error for unstructured noise).** When too few levels force rounding, the error is **correlated** → a smooth gradient breaks into **bands** the eye locks onto. **Dithering** spends that error as **high-frequency (blue) noise** instead so the local average tracks the true tone — the eye **integrates noise but sees structure**. [`fig-dithering-methods`]

• **stochastic quantization first** (the simplest form): round each pixel up/down **at random** with probability set by its position between levels → expected value exact, bands → white noise; this is **stochastic rounding** (unbiased low-precision NN training). No memory → noisier than needed.

• **error diffusion** gives the rounding *memory*: Floyd–Steinberg [@floyd-steinberg-1976] pushes each pixel's residual onto not-yet-visited neighbors → far less noise; **Ostromoukhov** [@ostromoukhov-2001] (intensity-dependent serpentine coefficients + a touch of blue noise → cleanest). Cleanest at **1 bit** (no interior levels to seam).

• ⚠️ **diffuse the error in LINEAR light**, not code values — the eye averages light, so only then does the dither's average brightness match the original (gamma/code-space dithering lifts the mid-tones; Jensen on the display curve).

• **not just for the display:** the same scatter-the-error trick recurs in **coded imaging** (binary/few-level masks), **computational illumination**, blue-noise sampling, and **stochastic rounding** in low-precision arithmetic. Registers **L2.52** here.

say clearly when things are encoded, when things are decoded.

KEY framing: a bare array of numbers is meaningless without knowing its encoding / color space

Discuss that different aspects of comp photo call for different encoding: linear (e.g. for dealing with physics, white balance, radiometry) vs. gamma vs. log

• rehash gamma and other encoding, ref to early section/chapter

• JPEG/film are usually some weird remapped versions for beautification

• as a result, you have more radiometric approaches to image editing/processing (starting with calibrated/linear/RAW values), vs older approaches that are more hand wavy because values were less clear

• the same three letters hide real differences: **channel order** (RGB vs BGR — OpenCV), **color space / primaries** (sRGB vs Adobe RGB vs Display P3 → Color technology), **encoding** (linear vs gamma), **range** (`[0,1]` float vs `[0,255]` uint8 vs 10/12-bit), and **premultiplied vs straight alpha**. A bare array doesn't say which — read the metadata / profile, don't assume.

• ⤳ **moved:** the draft for this is now in [[File formats and compression#Data versus metadata: EXIF]] (merged out of *Image representation* so all the metadata/EXIF material sits together); the notes below are the plan of record.

• an image carries more than the pixel array: **basic metadata** — pixel **width** and **height**, channel count, bit depth, the file **name / path**, creation date

• richer **EXIF** capture metadata written by the camera: **ISO, shutter speed, aperture, focal length**, lens, timestamp, **GPS**, **orientation** flag, embedded ICC profile

• ⚠️ **EXIF is frustrating in practice**: not every camera writes every field and the set / format is **inconsistent** across makers; some values (notably **ISO** and **shutter speed**) are **approximate / rounded** and **not radiometrically correct** — don't treat them as calibrated measurements

• **concrete numbers** (so the warning isn't hand-waving): the recorded **shutter speed is the *nominal* one** — a phone that actually exposed for 1/3.0 s or 1/3.9 s may write **"1/3 s" in both cases**, a photometric error of up to **~30 %** (the kind of EXIF exposure-time rounding documented in consumer-camera calibration work, [@burggraaff-etal-2019]); and the **EXIF ISO is the manufacturer's *setting*, not the measured sensitivity** — independent RAW measurements (DxOMark, ISO 12232 saturation method) routinely find the **real ISO differs from the marked one by up to ~⅓–1 stop** (a body set to "ISO 100" may measure ~80, "ISO 3200" may measure ~2000), because makers tune the gain/headroom and the standard leaves slack. So for any **radiometric** use (HDR weights, noise modeling, exposure fusion) treat EXIF shutter/ISO as a *hint*, and prefer a per-body **calibration** or the **DNG `NoiseProfile`** when available. (Refs: [@burggraaff-etal-2019]; DxOMark *ISO speed* glossary + sensor protocol; ISO 12232.)

• a catalog of the typical fields lives in the appendix → [[Common EXIF fields]]; the file-format side of metadata is in [[File formats and compression#Data versus metadata: EXIF|File formats]]

• 💡 **what metadata *should* travel with an image** (a recurring wish in this book): beyond the basics, a sane representation would carry exactly the things a bare array can't imply — the **encoding** (linear / gamma / log), the **color space / primaries** (ideally an embedded **ICC** profile), the **noise level** (the read + shot noise parameters, so downstream code knows the per-pixel uncertainty), and — when known — the **point-spread function (PSF)** (for deconvolution / deblurring). Most formats carry the first two unreliably and the last two not at all, yet later algorithms (white balance, **denoising**, **deblurring**, HDR) silently *need* this — so an honest image is pixels **plus** this provenance.

• 💡 **noise level deserves to be metadata — and almost never is.** Of that wish-list, the **noise level** is the most useful and the most neglected: a tiny per-sensor calibration (read noise + a shot-noise/gain term, i.e. the variance as a function of signal at the current ISO) tells downstream code **how much to trust each pixel**. It is **rarely written into any image file**, yet **denoising, HDR merging, deblurring, and any weighted least-squares fit** all silently need it — without it they fall back to guessing the noise from the image. It costs only a couple of numbers to store, so carrying it alongside the pixels would make a lot of estimation principled instead of ad hoc. (**DNG**'s `NoiseProfile` tag is a rare format that actually does this.) → [[Noise, signal-to-noise ratio and dynamic range]], [[Slopendium/Denoising]].

• preview the taxonomy of image operations by what they touch (point / domain / neighborhood) — developed in Point operations

packing multidimensional arrays in memory. Various options.

Which one we choose (different in Python and our C++ implementation)

• channel order: RGB vs BGR (e.g. OpenCV)

• interleaved (HWC) vs planar (CHW); CHW is what PyTorch wants later

• **the C++ image data structure** (C++ profile — show this as the book's working representation): a minimal `Image` holding `width, height, channels` and a flat `std::vector<float> data` in **planar** order, indexed `data[c*width*height + y*width + x]`, with a `float& operator()(int x, int y, int c)` accessor hiding the index arithmetic. (Python profile: a NumPy `H×W×C` array.)

origin top-left (y down) vs bottom-left (y up)

pixel center vs pixel corner: the half-pixel offset — a classic bug source; matters for resampling / warping (where-from vs where-to)

• **the problem**: a `float[H][W][3]` is **mute** — it does not say whether it is **linear or gamma**, **sRGB or P3**, `[0,1]` or `[0,255]`, **straight or premultiplied** alpha. Nearly every "why are my colors wrong?" bug is an operation applied to data in the **wrong type** (e.g. white-balancing or blending gamma-encoded values as if they were linear). The cure is to make the image **typed**.

• **a good image type encodes its gamma (and the rest of its provenance)**: alongside the array, a small **type record / tags** — **encoding** (linear / gamma / log), **color space / primaries** (ideally an embedded **ICC** profile), **range and bit-depth**, **alpha kind** (straight vs premultiplied). The encoding is **part of the value, not a fact you keep in your head**. (This is the "what metadata *should* travel with an image" wish from *Beyond the pixels*, promoted into the type.)

• **conversion = cast = convert the numbers AND re-tag** (the key discipline): a `to_linear()` must **divide out the gamma *and* set the encoding field to `linear`**; a `to_uint8()` must **scale and round *and* set the range / bit-depth tag**; a color-space convert applies the 3×3 *and* updates the **primaries** tag; (un)premultiply flips the **alpha** tag. A cast that changes the numbers **without** updating the tag (or the reverse) is exactly the bug. Done right, **type and data are always consistent**, and an operation can **refuse, or silently auto-convert,** a mismatched input.

• **why it's worth the ceremony**: it makes the **illegal states unrepresentable**. White balance and exposure can *insist* on **linear** input (multiply-in-linear, [[Measuring and encoding color|Color technology]]); a blend can *insist* on matching **alpha kind** and **color space**; a save can *insist* on an output **encoding**. A type check catches the classic mistakes up front, instead of the *viewer* catching them later. It is **units for images**.

• **in practice**: most arrays are untyped (NumPy, OpenCV) — which is *why* these bugs are everywhere; but **color-managed** apps, Adobe's pipeline, and **DNG-style** containers ([[Appendices#DNG: the Digital Negative]]) effectively carry this type, and the book's own `imageops` treats encoding/space explicitly. The takeaway: **track the type, and let conversions move it — don't keep it in your head.**

• **RGBA**: a 4th **alpha** channel = per-pixel opacity / coverage; **premultiplied** (RGB already scaled by α) vs **straight** alpha is a classic bug — full treatment in **Compositing**

• other extra channels: a depth / Z channel, a matte, or **hyperspectral** (many narrow bands) — the array just gains channels

• **video** = an image sequence: just an extra **time** axis (H×W×C×T); most per-frame operations generalize directly (and motion couples frames — Motion/Video)

• **higher-dimensional** imaging: volumetric **3D** (medical CT/MRI: H×W×D), **4D light fields** (two spatial + two angular), spectral cubes — the same array machinery, more axes

• wrap pixel access in a **safe accessor** so out-of-bounds reads don't crash or read garbage — and decide what a pixel *past the boundary* means:

• **black / zero** (pad with 0 — darkens edges), **clamp / replicate** (repeat the edge pixel), **mirror / reflect** (fold the image back), **wrap / tile** (periodic — what the DFT assumes)

• the choice matters for **convolution, resampling and warping** at borders; mirror is the usual safe default [edge-handling modes figure]

• 🔁 **quick reminder** (already previewed back in *Image representation → Three kinds of operation*): image operations split by **what they touch** — **point** (range; same pixel's *value* → one 1-D curve), **domain** (a *different* location → resampling/warping), **neighborhood** (a *window* → convolution/bilateral). No need to re-derive it here — this chapter just **zooms into the point branch**, the simplest, and where we start. [the three operation types figure, with the **point-operation** branch **highlighted**]

• the **histogram** (the distribution of values — its own chapter, next) is the natural companion to point ops

• develop everything on a grayscale value; for color, apply the same curve per channel (saturation is the exception)

for single channel : simple 1D input output curve. the whole operation *is* that curve (the editor's curves tool); identity = the 45° diagonal

one big question: in which units (linear, gamma, log)

Say that traditional photography has always involved controlling the mapping from scene radiance to output photo tone.

side bar: the zone system. Zones are intensity ranges. encourages photographers to think/act on mapping from scene to negative and then from negative to print. Sometimes in a local way. Lots of tools: exposure, type of film, pre-exposure, chemistry, darkroom.

• 💡 **Big lesson:** **additive vs multiplicative → choice of encoding** — ask whether a tonal op is additive (→ linear/display offset) or multiplicative (→ linear light, a sum in log); light and perception are multiplicative

• exposure is multiplicative in linear space ($L_\text{out} = 2^s L_\text{in}$, one stop = a factor of 2)

• the +1-stop curve bows upward in display space; highlights clip at 1 (unrecoverable)

• do it in linear light; the naive gamma-space multiply clips at input ½

• brightness = additive offset $L_\text{out}=L_\text{in}+b$; lifts blacks, fogs the shadows

• recommend against brightness; exposure is more radiometric (keeps black at black). Brightness survives as a legacy slider / quick shadow lift

• contrast = steepen about a pivot $L_\text{out}=g(L_\text{in}-p)+p$; gain >1 steepens and clips both ends

• big question: in linear, gamma, or log (three legitimate looks; defer the full argument to tone mapping)

• sidebar: the S-curve (sigmoid) — roll off the extremes instead of clipping

• **levels**: set the **black point** and **white point** (+ a mid-gray **gamma**) — a piecewise remap that stretches the used range to [0,1]; the workhorse first edit

• read them off the **histogram**: clip the empty ends, pull the mids with the gamma slider

• a free-form **tone curve**: drag control points, smoothed by a **spline** kept **monotone** (Catmull-Rom / monotone cubic) so tones never invert [tone-curve / spline figure]

• **sidebar — splines (the math)**: fit a smooth curve through the control points (knots) $(x_i,y_i)$ as a **piecewise cubic** — one cubic per interval, the pieces joined for continuity.

• **cubic Hermite** form on $[x_i,x_{i+1}]$ with $t=\frac{x-x_i}{x_{i+1}-x_i}\in[0,1]$: $p(t)=h_{00}(t)\,y_i+h_{10}(t)\,m_i+h_{01}(t)\,y_{i+1}+h_{11}(t)\,m_{i+1}$, with basis $h_{00}=2t^3-3t^2+1,\ h_{10}=t^3-2t^2+t,\ h_{01}=-2t^3+3t^2,\ h_{11}=t^3-t^2$. The knot **tangents** $m_i$ are the free choice — picking them is what separates the schemes.

• **Catmull–Rom** (interpolating, $C^1$): tangent = centered difference $m_i=\frac{y_{i+1}-y_{i-1}}{x_{i+1}-x_{i-1}}$ — local and cheap, but can **overshoot** (so it may go non-monotone).

• **natural cubic** ($C^2$): require continuous second derivatives and minimize total curvature $\int (p'')^2$, which gives a **tridiagonal** linear system for the knot slopes — smoothest, but still can overshoot.

• **monotone cubic** (Fritsch–Carlson): start from Hermite tangents, then **clamp** them so every segment is monotone — if a secant slope $\Delta_i=\frac{y_{i+1}-y_i}{x_{i+1}-x_i}=0$ set the neighboring tangents to $0$; otherwise keep $(\alpha,\beta)=(m_i/\Delta_i,\ m_{i+1}/\Delta_i)$ inside the circle $\alpha^2+\beta^2\le 9$. This **guarantees no inversion**, which a **tone curve** requires (brighter in $\Rightarrow$ brighter out). [tone-curve / spline figure]

• **B-splines** are the *approximating* cousin — the curve follows a control polygon and need **not** pass through the points; smoother and easy to keep monotone, at the cost of exact interpolation.

• subsumes exposure / contrast / levels; container for the principled tone-mapping curves (next chapter)

• **sidebar — how analog photography shaped the same mapping**: long before a digital curve, photographers controlled the **scene-tone → printed-tone** mapping by physical and chemical means —

• **exposure**: where you *place* a scene luminance on the film's response (the Zone System's "place this shadow on Zone III, let the highlights fall where they may")

• the film's **characteristic (H&D) curve** and its **gamma / contrast** — chosen by emulsion *and* by **development** (time / temperature / agitation / developer; **push / pull**; "**N / N+1 / N−1**" development to expand or compress the scene's range onto the film)

• **pre-exposure / flashing**: a uniform sub-threshold exposure that lifts deep shadows above the film's *toe* (raising the floor without touching highlights)

• at print time: the **paper grade** (contrast grade 0–5), the print developer, and local **dodging & burning** (hold back / add light per region)

• the free-form tone curve (and the per-region masks of *Local point operations*, below) is the **digital descendant** of this whole toolkit. *Cite **Ansel Adams**' trilogy — **The Camera**, **The Negative**, **The Print** — and the **Zone System** (developed in *The Negative*).*

• the **basic color edits** every editor exposes — the color analog of the tone curve. **This is the home for the implementable mechanics** (the equations below); the *perceptual-space* nuance — which space to scale chroma in, why skin needs protecting — is the job of [[Measuring and encoding color|Color technology]] (don't defer the equation there)

• **saturation**: split each pixel into **luminance** $Y$ and **chroma** $(\text{RGB}-Y)$, scale the chroma, recombine — $\text{RGB}' = Y + s\,(\text{RGB}-Y)$ (so $s=0$→gray, $s=1$→identity, $s>1$→more saturated; equivalently $(1-s)\,Y + s\,\text{RGB}$). A *uniform* $s$ over-saturates already-vivid colors and skin

• **vibrance** (implementable): the same operation but with a **per-pixel gain** that fades as the pixel's own saturation rises, and protects skin — replace the constant $s$ by

$$g = 1 + v\,(1-\text{sat})\,w_\text{skin}(\text{hue}), \qquad \text{RGB}' = Y + g\,(\text{RGB}-Y)$$

where **sat** is the pixel's current saturation (e.g. HSV $S = \tfrac{\max-\min}{\max}$, or chroma magnitude / luma), **v** the vibrance amount, and **w_skin(hue)** a hue weight $\approx 0$ on skin / orange hues and $1$ elsewhere — so muted colors get the full boost while vivid colors and skin barely move (set $w_\text{skin}\equiv 1$ for the skin-agnostic version)

• which space to scale chroma in (perceptually uniform), and why skin needs protecting — deferred to [[Measuring and encoding color|Color technology]]

• **LUT (look-up table)**: bake any color/tone mapping into a precomputed table indexed by pixel value, so applying it is one **lookup per pixel** — fast, and the standard way to ship a *look*

• **1-D LUT** = three independent per-channel curves (exposure / contrast / levels / gamma collapse into one); **3-D LUT** = a full RGB→RGB map that handles **cross-channel** effects a 1-D LUT can't (white balance, film/grade looks — the `.cube` files of video), at the cost of a coarser sampled grid + interpolation

• LUTs are also a **speed** trick: precompute an expensive per-value function once, then index it (ties to fixed-point / 8-bit pipelines)

• a **separate craft** from stills / radiometric editing — the *look* of motion pictures, with its own tools and vocabulary; less radiometric, more color craft

• the colorist's primaries: **lift / gamma / gain** wheels (shadows / midtones / highlights, each with a color offset); **primary** (whole-frame) vs **secondary** (qualified by hue / luma, or masked) corrections

• the working space is usually a camera **log** encoding (S-Log, Log-C) under a color-management pipeline (**ACES**); the finished *look* is shipped as a **3-D LUT** (`.cube`), applied per frame (ties to [[#Lookup tables]])

• the **teal-and-orange** look: push skin into warm **orange**, everything else toward its **teal** complement — complementary contrast that makes people pop; popularized by ***O Brother, Where Art Thou?*** (2000 — among the first features with a full **digital intermediate**; Coen brothers, cinematographer Roger Deakins)

• the **stills-side cousin**: Lightroom's HSL / color-mixer and color-grading panels

• ties: [[#Lookup tables]] (3-D LUTs are the delivery format); [[Measuring and encoding color|Color technology]] (color management, log / ACES); tone mapping (the dynamic-range side)

• strictly not a point operation: apply a tool through a **selection mask** (the result depends on *where*, not just the pixel's value)

• **selections**: by value / color (magic wand), by **gradient** (linear / radial), by **brush**, **edge-aware** (snap to edges → Big lesson: edge-preserving = affinity), and increasingly **AI / semantic** (segment-anything-style)

• 💡 **the moment a point operation goes *local*, it raises the central question of "*which pixels?*" — segmentation and matting.** A hard binary **selection** is a *segmentation*; a soft-edged one (feathering, hair, motion blur, partial coverage) is a **matte** (a per-pixel α, exactly the alpha channel of [[#Alpha and extra channels]]). So even a humble masked exposure tweak is the front door to a deep topic — how to decide the mask, and how to make its border *soft and correct*. **Forward ref → [[Compositing, segmentation and matting]]** (the full treatment: matting equation, trimaps, alpha estimation), with edge-aware mask refinement in [[Guided image filtering]] / [[Bilateral filtering]].

• the bridge to **local tone mapping** and edge-preserving filtering (next chapters)

• a radiometrically-correct scene-linear render looks **dull and ~1–1.6 stops dark**; nothing is broken — it just lacks the camera/raw "look"

• KEY: do processing in scene-linear; apply the **beauty curve only at the very end, for display**; never bake it into data you compute on

• mid-gray is ~0.18 in linear, not 0.5 → honest encode lands midtones low; and a bare encode has no highlight shoulder

• the missing midtone lift measured vs a Lightroom JPG ≈ 1–1.6 stops

• tone mapping compresses an enormous range and is rare in practice; the beauty curve renders **every** standard-range photo; shapes rhyme, jobs differ

• smooth, monotone, bows above the diagonal (midtone lift) with a soft highlight shoulder; must be smooth (monotone cubic) — kinks/plateaus band the highlights

• real shape lives in the camera profile (Adobe baseline tone curve + HueSatMap + LookTable); we recover an equivalent baseline curve empirically (fit linear-render luma → developed-JPG luma)

• hue-preserving (scale all channels by the luma ratio) keeps highlights colored → magenta/cyan sky near the clip

• fix: **highlight-weighted hybrid** — hue-preserving in shadows/mids (keeps deep-blue sky saturated), blend to **per-channel** near white so highlights bleach cleanly to white; ease saturation off in the highlights too

• a gentle (~+10%) luma-preserving saturation finishes it; HueSatMap/LookTable are *style*, not necessity, and easy to get wrong (purple blues, garish greens) — stop at baseline curve + highlight-aware saturation

• process in linear; beautify last — Poisson/HDR/WB/deblur must see scene-linear, not the look-curved render

• normal color science aims the render at the **scene**; viewers judge it against **memory** — and Bartleson measured that remembered colors of familiar objects (sky, grass, skin) are **more saturated and more prototypical** than the real distributions

• consequence: a colorimetrically faithful pipeline reads as slightly wrong — the standard complaint is **skies too magenta** (though careful naked-eye sky is more magenta than people believe; preference, not accuracy, is the target)

• every vendor pipeline (Canon, Nikon, Fujifilm, Adobe) therefore ships deliberate, hue-dependent distortions for a handful of **memory colors**; this chapter builds that stage at the bottom rung of a complexity hierarchy: a **pure color point operation** — the pixel's own color decides everything, no spatial or semantic information (classifiers and masks come later, in the local-adjustments world)

• preference does not climb forever with saturation; it is an **inverted U**. Nudge a color toward its memory prototype and it looks better, push past a ceiling and vivid reads as fake, and preference drops (Yendrikhovskij, Blommaert & de Ridder 1999: reproduction is judged best just *above* the natural saturation distribution, worse beyond it — the **naturalness constraint**)

• so every move here is deliberately **small and bounded**: a few degrees of hue, a modest saturation gain, kept under the naturalness ceiling. The failure mode is not "too accurate," it is the over-cooked HDR/filter look — memory colors overshot into garish

• this is why skin moves least, why the house defaults are single-digit-degree nudges, and why the sibling beauty curve stops at a gentle ~+10% saturation ([[#Beauty curves: the camera "look"]]) — restraint is the craft, not the size of the slider

• preferred is also **not universal**: a tuned, per-vendor, partly cultural choice (a Fuji film-sim vs a muted cinema look), fixed by **preference experiments** (paired comparison, expert-retouched sets like MIT-Adobe FiveK), not derived from colorimetry

• a **corridor** = a soft bump over hue, weight w(h) = smoothstep(1 − |h − h*|/W), tapered by saturation and value (ramp to full over s ∈ [0, 0.12], v ∈ [0, 0.10]) so gray and near-black are immune; display-referred HSV, hue in degrees

• **sky blues**: center h* = 215°, half-width 55° (reaches 160–270°); hue slider full-scale ±20°, house default a ~6° nudge toward cyan; saturation ±50%

• **vegetation greens**: center h* = 100°, half-width 50° (50–150°); ±20° full-scale — the usual move is away from yellow, slightly deeper

• **skin oranges**: center h* = 25°, half-width 28° — deliberately the tightest corridor and the smallest range (±10°): skin is where errors offend most, so the default moves skin **least**

• **translation**: add Δh inside the corridor — the whole region shifts (sky toward cyan). On a hue-vs-hue plot: height

• **contraction**: add k·(h* − h) inside the corridor, 0 < k < 1 — a restoring pull toward the preferred hue; the hue **distribution narrows**. On the plot: negative slope through the anchor; |slope| < 1 guarantees it contracts without overshoot

• contraction is the important half for **skin**: viewers object to blotchiness (hue variance) far more than to a uniform cast, so the preferred move is uniformity, not tint — Bartleson again: memory colors are narrower than reality

• saturation gets the same pair: scale it, or compress its spread toward a preferred value ("clean skin")

• NTSC (1953) chose the **I axis** of YIQ to lie along the skin line — skin from all complexions clusters startlingly tightly in chromaticity, along the orange–cyan direction — and allotted I the most chroma bandwidth (Wintringham 1951 for the colorimetry)

• receiver makers then shipped **automatic tint / flesh-tone correctors**: circuits that rotate hues near the I axis toward it — a literal hue contraction in hardware, decades before digital pipelines, motivated by transmission phase errors landing on the one color viewers cannot forgive

• the modern descendants: per-camera **ProfileHueSatMap / LookTable** lattices in DNG profiles (the same operation, tabulated — plot Δh against h and vendor tables show restoring slope around sky and skin), colorist hue-vs-hue curves with a flat spot through skin, phone-ISP skin uniformity

• a pure color point operation cannot tell a blue car from the sky or a wooden wall from a face; both get the corridor treatment

• that failure is the *motivation* for the next rung: gate the same deltas by **what the pixel is**, not only what color it has

• the operation stays the corridor delta; a per-pixel **semantic weight** multiplies it: `delta(pixel) = corridor(h, s, v) × P(class | pixel)` — color decides *what* the move is, a detector decides *where* it applies. The blue car gets P(sky) ≈ 0 and keeps its paint

• **detection is a segmentation problem**, and the practical recipe is: a compact scene-segmentation network (the book's raw developer runs SegFormer-B0 trained on ADE20K, 150 classes aggregated to three maps: sky = {sky}; vegetation = {tree, grass, plant, flower, palm}; skin = {person}), run **once per photo at low resolution** — 512² in, stride-4 probability maps of 128² out — then **sampled with smooth (bilinear) upsampling** at whatever resolution renders. Full-resolution inference buys nothing: the network cannot produce edges finer than its output stride, and the corridor taper already feathers the transition

• consequences of the low-res maps, stated plainly: boundaries carry ~1% -of-frame softness, invisible under gentle color moves, visible under hard local adjustments; the standard cure is a **guided (joint-bilateral) upsample** against the full-resolution image, which snaps the soft boundary to real edges

• **soft probabilities, not hard labels**: the weight is P(class), never argmax — misclassifications then fail gently (a half-confident pixel gets half the move), and the blend feathers itself

• classic pre-network detectors are worth naming because they ARE the corridors: chromaticity corridors plus spatial priors (sky is blue *and up*, skin is orange *and near a face detection) — the rung-2 machinery with position added. The network replaces hand-drawn priors with learned ones; the interface (a per-pixel weight in [0,1]) is unchanged

• the same maps immediately give **semantic local adjustments** (Lightroom's Select Sky / Select Subject): the mask is the weight map itself, any slider rides it — one mechanism, two consumers (the automatic look and the user's masks)

• ⚠️ and the "beautify last" discipline from the beauty-curve chapter applies unchanged: these distortions are for display; never compute on beautified data

• the **histogram** = the distribution of pixel values; reading it tells you exposure, clipping, contrast, and where the tones sit [histogram figure]

• the on-screen **RGB and/or luminance histogram** in the camera and in editing apps; a **live histogram** in the viewfinder / on the rear screen while composing

• **clipping warnings** ("**blinkies**"): blown highlights and crushed shadows flash to flag lost detail at each end

• ⚠️ the displayed histogram is typically of the **gamma / display-encoded** image, **not** the linear data — so its shape (and where mid-gray sits) reflects the encoding, not scene radiance (ties to *depends on the encoding space*, below)

• beyond per-channel: a **3-D color histogram** = the distribution of pixels in **color space** (a cloud / occupancy in the color volume)

• its shape **depends on the color space** you build it in — RGB cube vs a perceptual space (Lab / opponent) bin the same pixels very differently

• used for **palette extraction, color transfer**, and **segmentation** (clusters in color space)

• its **shape depends on the encoding space** — the *same* image has a very different histogram in linear vs gamma vs log [histogram-encoding-spaces figure: linear piles into shadows, gamma spreads it, log spreads shadows more; 18% gray lands at 0.18 / ~0.46 / ~0.75]

• warning: it's a **sampled / discretized** estimate — spikes and gaps are artifacts of **binning / quantization**, not the scene

• **histogram equalization**: use the image's **cumulative histogram (CDF)** as the tone curve → flattens the histogram, maximizing global contrast (often too aggressively) [histogram-equalization figure: the CDF *is* the transfer curve]

• pseudocode: bin → running-total CDF scan → normalize → apply as a per-pixel LUT

• **histogram matching** (a.k.a. histogram **specification**) generalizes equalization: instead of flattening to a *uniform* target, reshape an image's histogram to match **any target distribution** — a chosen shape, or *another image's* histogram

• **the recipe = compose two CDFs**: equalize the source (push it through its own CDF, $\text{CDF}_\text{src}$, to a uniform), then **"un-equalize" through the target's inverse CDF** — $L_\text{out} = \text{CDF}_\text{tgt}^{-1}\big(\text{CDF}_\text{src}(L_\text{in})\big)$; both are 1-D LUTs, so matching is still a single **point operation** [histogram-matching figure: source histogram + target histogram → matched result + the composed transfer curve]

• **equalization is the special case** where the target is uniform ($\text{CDF}_\text{tgt}=$ identity) — so matching, equalization, and a hand-drawn levels curve are one family

• **for color**: matching each channel independently **shifts hues** (the channels aren't independent), so do it in a **decorrelated / perceptual** space, or use the proper multi-dimensional version — **color transfer** (Reinhard et al. 2001: match the **mean and standard deviation** per channel in the decorrelated $l\alpha\beta$ space), with the exact N-D form via **optimal transport** (Pitié et al.)

• 💡 forward ref: histogram / color matching is the workhorse behind **harmonization** (make a pasted region's tones match the destination — Compositing), **color grading / style transfer** (impose a reference look), and **texture synthesis** (match value statistics) — all instances of "make this image's statistics look like that one's"

• refs: Reinhard, Ashikhmin, Gooch & Shirley 2001 (color transfer between images); Pitié, Kokaram & Dahyot 2007 (automated N-D color transfer / optimal transport); Gonzalez & Woods (histogram specification — textbook)

• **Greg Ward's constrained / histogram-based tone mapping** (1997): equalize, but **cap the local slope** so contrast is never exaggerated beyond what the eye accepts — a perceptually-bounded version for HDR [Ward et al. 1997]

• unsharp mask: detail $=\text{in}-\text{blur}$, add it back amplified; flat regions untouched, edges/texture boosted; the whole operation is one $\delta-g$ convolution (center-positive, surround-negative). [`fig-unsharp-mask`, `fig-sharpen-kernel`]

• the darkroom relic: sandwiching a negative with a slightly **blurred, low-contrast positive** ("unsharp mask") boosts edge contrast where the two disagree — exactly $\text{in}-\text{blur}$, done in silver. (Moved here from the in-line sidebar; now its own short section, between *Linear sharpening* and *Non-Linear Sharpening*.)

• the unsharp mask is a weighted **difference** (in − blur), and a blur is an **average**, so the result depends on the **space** you compute it in — same as tone mapping / color. Apply the same mask in **gamma** (stored values, the editor default), **linear** (physical light), or **log** (ratios) and you get three different looks, most visible in the **edge halos**: linear → strong bright halo / weak dark; gamma → perceptually even; log → gentle in highlights, firm in shadows. No single right answer; worth seeing. [`fig-sharpen-space`]

• linear unsharp masking has two failure modes: it **amplifies noise** (in flat areas the high frequencies are mostly noise) and **haloes** at strong edges. [`fig-oversharpen-halo`]

• **adaptive / non-linear** unsharp masking **gates** the boost — sharpen **more in detailed / edge areas, less in smooth ones** (by local **edgeness** or **intensity**); cubic / rational variants (Ramponi) and adaptive control (Polesel et al.); apply mostly to **luminance**, in **midtones**, near **edges** — the principled version is the edge-preserving part. [`fig-sharpen-demo`]

• forward-ref: pushed past *amplifying* surviving detail into *inventing* it is **super-resolution** ([[What can super-resolution actually do?]]); the edge-aware "gate by who-belongs-together" is the **affinity** idea developed in [[Bilateral filtering]].

• **the AE problem**: pick aperture/shutter/ISO so the scene lands at the **middle-gray** (~18%) target; the catch is *which* pixels are representative

• **average / center-weighted** metering — average the frame (fooled by bright skies → silhouettes, snow → gray); center-weighting assumes a centered subject

• **matrix / evaluative / pattern** metering — split into **zones**, combine brightness + contrast + focus point (+ color + lens distance = Nikon's **3D Color Matrix**); historically a lookup against a database of reference scenes; the default today

• **spot** metering — meter one small zone; the photographer chooses the anchored tone (Adams's Zone System)

• **learned** metering — a small network (often + semantic segmentation: sky/face/backlight) predicts the photographer's exposure; phones choose exposure jointly with burst/HDR (deliberately under-expose, recover shadows by stacking → Multiple exposure), leaning on the affine+truncated noise model

• 💡 same arc as AWB below: one hand-coded statistic → multi-zone pattern model → learned estimator, all guessing intent from pixels

• spectral perspective

• Von Kries

• **white balance is a per-channel multiply — but only correct in *linear light***: it scales each channel by a gain (Von Kries), and a scaling **commutes with a *linear* map but not a non-linear one**. Apply the same gains to **gamma- / tone-curve-encoded (non-linear)** values and you get the *wrong* answer — the channel gains then **shift hue and change apparent saturation**, not just neutralize the cast. So white balance belongs **early, in the linear pipeline**, alongside exposure (also a multiply-in-linear-light). [fig-wb-linear-vs-nonlinear — the same white-balance gains applied in linear light vs. on gamma-encoded values: the hue/saturation shift] (→ [[#Linear vs gamma vs. log encoding]])

• **white balance from color temperature and tint (the raw-converter slider model)** — the two sliders just pick a **target illuminant white point**, and the diagonal gains follow:

1. **Temperature** $T$ (Kelvin) selects a point on the **Planckian (blackbody) locus** — its chromaticity $(x_c(T), y_c(T))$ from a standard closed-form fit (Kim et al. 2002): warm/low-K → orange, cool/high-K → blue.

2. **Tint** nudges that point **perpendicular to the locus**, along the **green↔magenta** axis (a signed $\Delta uv$ in CIE-1960 $uv$) — correcting casts a pure blackbody can't, e.g. fluorescent green.

3. Convert the white point to tristimulus: $X = x/y,\; Y = 1,\; Z = (1-x-y)/y$, then to the working/sensor space $\mathbf{W} = M_{XYZ\to RGB}\,(X,1,Z)$.

4. Set the **Von Kries gains** $k_c = G_w / W_c$ (normalize to green) so a surface under that illuminant reads **neutral**.

So "$T=5500$ K, tint $0$" and "$(k_R,k_G,k_B)$" are **two faces of the same diagonal correction** — the sliders are a human-friendly parameterization of the per-channel gains. (M lives in [[#Reproducing color]]; *where* the multiply must happen is [[#Linear vs gamma vs. log encoding]].)

• old style: gray assumption, bright pixels

• 3D matrix regression style

• modern ML solutions

• **skin tones** — skin is a **memory color** cameras are tuned around (face detection feeds the meter + AWB so the *person* lands right); but that tuning has a **history of bias** (the **Shirley card** calibrated film/AE/AWB to light skin, rendering darker skin poorly). Obligation: design & test AE/WB across the full range of skin tones → ethics in [[Ethics of computational photography]]

• multiple / mixed illuminants: one global per-channel gain can't fix a scene lit by two light colors (forward ref: multi-light WB, Hsu et al.)

• metamerism failure: WB can't undo a bad illuminant spectrum (narrow-band fluorescent / cheap LED)

• Color Rendering Index (CRI): how faithfully a source renders colors vs a reference illuminant

uncompressed/raw → lossless (PNG, TIFF) → lossy (JPEG)

lossless = exploit statistical redundancy; lossy = also discard perceptually unimportant info (ties to perception chapter)

• stuff like name, image size

• capture settings (ISO, shutter, aperture, focal length), GPS

• orientation flag (the infamous "why is my image sideways" bug)

• ICC color profile (ties to color management)

• typical-fields **catalog → appendix [[Common EXIF fields]]**; the practical caveats (coverage incomplete & inconsistent; ISO / shutter approximate & not radiometric) are introduced in [[#Image representation]]

-Fun with EXIF: my kids growing up, photos from X years ago

• we use PNG because it's easy to read

• mention alphas in passing to warn people (because they need png with no alpha)

Central idea : leverage human perception (cite early chapter) to adjust the number of bits to what matters. Remind people of CSF and different color sensitivity

• **framing**: JPEG is a perfect illustration that **perceptual criteria can be made quantitative and objective** — "what the eye won't miss" becomes a concrete **quantization table** and bit allocation rather than hand-waving; perception turned into numbers you can compute, tune (the quality knob), and measure

• the **quality knob** = a multiplier on the quantization table; show a **quality sweep** of the same photo (q ≈ 85 → 5) with the resulting file size under each — degradation (softening → blocking → color bleed) grows as the file shrinks [JPEG quality-levels figure]

#### Stage 1: opponent color

Opponent color space

#### Stage 2: chroma subsampling

chroma subsampling (4:2:0 / 4:2:2): coarser chroma than luma because color CSF is lower

#### Stage 3: the 8×8 DCT

DCT

#### Stage 4: quantization — the lossy step

quantization table

• aside — **8-bit pixels carry every DCT coefficient to ≈8 bits, frequency-independent** [`fig-dct-8bit-roundtrip`]: render one basis coefficient → quantize pixels to 8 bits → transform back; recovered ≈ intended to ~8 bits for low/mid/high frequency alike, because the DCT is an orthonormal rotation (white pixel-rounding noise spreads evenly over all 64 coefficients). So JPEG's coarse high-frequency quantization is a **perceptual** choice (CSF) + energy statistics, *not* a pixel-storage limit.

#### Stage 5: entropy coding

classic entropy coding, with smart block order

#### Limitations and artifacts

limitation: 8-bit only (motivates other formats); artifacts: blocking, ringing

• **generation loss is largely overblown:** re-saving at a fixed quality settles into a **fixed point** (coefficient quantization ≈ idempotent) and stops degrading — same resilience as the repeated-quantization experiment of 3.1 (in theory it compounds, in practice it's stable). [`fig-jpeg-generation-loss`]

• **combined worst case** — the §3.1 iterated-exposure loop *with* a JPEG round-trip at every step: 8-bit banding and DCT blocking/ringing compound together, still tending to a fixed point. [`fig-exposure-jpeg-roundtrip`]

• **exception — deblock-on-load drift:** software that prettifies low-quality JPEGs on load (Photoshop's deblocking) breaks the fixed point → the load→save loop can **drift** [@hackerfactor-himym] (▶ youtube Fk6kV5N1rzs); a filter-free codec does *not* drift (▶ youtube gJJachY651c).

#### Removing the artifacts: JPEG deblocking

• post-hoc **deblocking / artifact reduction**: POCS onto quantization intervals (Zakhor 1992 [@zakhor-1992]); shape-adaptive DCT (SA-DCT, Foi et al. 2007 [@foi-etal-2007]); in-loop deblocking in video codecs (H.264, List et al. 2003 [@list-etal-2003]); learned CNNs (AR-CNN, Dong et al. 2015 [@dong-etal-2015]). All trade fidelity for appearance → decode is no longer a faithful inverse of encode (the drift above).

usually before demosaicking

linear

proprietary vs DNG

• **"RAW" is not always purely raw**: many cameras pre-cook it to some degree — black-level subtraction, defective/hot-pixel correction, sometimes lens-shading/vignetting correction or even mild denoising; some "RAW" is lossy-compressed or non-linearly encoded. So RAW is a *spectrum* of how cooked it is, not literally the untouched sensor readout.

• tools/libraries to read RAW: **dcraw** (the classic), **LibRaw** (C++, dcraw's successor) and its Python wrapper **rawpy**; open-source developers **darktable** / **RawTherapee**; **Adobe DNG Converter / DNG SDK**; **exiftool** for metadata; commercial **Lightroom / Capture One**.

quickly in passing

EXR (OpenEXR), Radiance .hdr, 16-bit TIFF — needed to store HDR output (see HDR chapter)

HEIC/HEIF (Apple default now, 10-bit / HDR-capable), WebP, AVIF, JPEG XL

• **the idea**: replace the hand-designed transform + quantizer + entropy coder of JPEG with a **trained autoencoder** — an **analysis** network maps the image to a latent, the latent is quantized and entropy-coded under a **learned prior**, a **synthesis** network reconstructs. The whole pipeline is optimized **end-to-end** for a **rate–distortion** objective (bits vs. a distortion/perceptual loss), so the codec *learns* what to throw away.

• **why it matters**: at low bitrates learned codecs beat JPEG/even AVIF on perceptual metrics, and they can optimize for **perceptual / GAN** losses rather than MSE (sharper, more natural reconstructions at the same rate). They also generalize: **video**, **depth**, and **scientific** data, and the latents double as features for downstream tasks.

• **the catches**: heavy compute (a neural net to *decode*, not just encode), **non-determinism / version-coupling** across encoder–decoder, and no universal standard yet — though **JPEG AI** (ISO/IEC) is the first standardized learned image codec. Ties to [[Deep learning]] (autoencoders, entropy models) and to generative priors in [[Single-image computational photography]] (super-resolution, inpainting are the same rate–distortion-with-a-prior idea pushed to extreme compression).

• photoshop, layers

• mention alpha again in passing.

• the **image signal processor** runs a fixed **pipeline** from RAW to JPEG; the **order matters** — mostly **linear light** until the end [ISP block-diagram figure]

• **3A** (auto-exposure, auto-white-balance, auto-focus — AF in Optics); **black-level** subtraction; **hot / defective-pixel** correction

• **demosaick** (→ ch) · **denoise** (→ ch) · optional **lens corrections** (vignetting, CA, distortion)

• **white balance** + **color matrix** (sensor → standard RGB) · **tone mapping / curves** (incl. local) · **sharpening**

• **gamma encoding** · **beautification curves** · **JPEG** compression (→ ch)

• advanced: **aberration correction**, computational stages (HDR+, night mode) — later parts

• the throughline (Big lesson): do the physics (WB, demosaick, denoise) in **linear**, **encode (gamma) near the end**

• e.g. **Adobe Lightroom**: keep the **RAW** untouched and store edits as **parameters** (a recipe — exposure, contrast, curves, masks), recomputed on the fly

• benefits: **reversible, re-editable, resolution-independent**; needs a **fast (GPU) pipeline** to recompute interactively

• the parametric pipeline ≈ a programmable ISP exposed to the user