• The principle of image formation stayed largely unchanged since photography's invention: a lens focuses scene rays onto a 2D sensor that records them directly; the final image is essentially a *copy* of the optical image, and quality came from better **optics**.

• Computational photography breaks this: insert computation **between the optical image and the final picture** → alleviate physical limits, enable flexible post-capture editing, record new information (depth, structure), and create novel visual experiences.

• The key shift: the optical image **need not resemble** the final image, which vastly expands imaging strategies. Computation is not just post-processing of a normally-formed image — it changes the optics too: **new optics are co-designed with the computation** to optimize the whole pipeline.

• **coded optics, concretely**: in **coded-aperture** imaging (Levin et al. 2007) a patterned mask replaces the round aperture so the out-of-focus blur becomes *invertible* and its shape encodes depth — the raw photo looks oddly blurred, but mask and deblurring algorithm are **designed together** so that what you recover *after* computation is sharp and depth-aware (forward ref: coded aperture / computational imaging in [[Advanced computational photography]]).

• **"Computation is the new optics."** Optics long extended our ability to image the world; digital processing now vastly expands how we *form* and enhance images.

• Analogy (Ansel Adams, also a trained pianist): the negative is a *score*, the print a *performance*; a "straight print" reproduces the negative mechanically — computation lets us further **interpret the score**.

• Prettier pictures

• Alleviate physical limitations

• Record more info (depth, structure, layering).

• Facilitate human-driven post-processing

• New visual experience

• Reveal the invisible

• traditional, traditional with multiple exposures, new optics/change imaging

• (and now more and more **generative**).

• From generation (painting) to measurement/snapshot to generation again.

• **Sidebar: History of Photography**

See e.g. https://photographpreservationprogram.hsites.harvard.edu/harvards-history-photography-timeline-text-only

https://en.wikipedia.org/wiki/Timeline_of_photography_technology

• **timeline** (rough — to lay out as a clean chronology when writing):

• **optics first**: the **camera obscura** (known since antiquity; described by Alhazen ~1020; a drawing aid for Renaissance painters).

• **1826** Niépce — first permanent photograph (heliography); **1839** Daguerre (daguerreotype, *doesn't fade*) and Talbot (calotype — the negative→positive idea) — photography is announced.

• **1851** Archer — wet **collodion**, cutting exposure to a few seconds; **1861** **Maxwell** — first color photograph (three filters).

• **1888** Kodak roll film ("you press the button…") → **1900** Brownie — photography for everyone; **1895** the **Lumière** brothers (cinema; Autochrome color 1907).

• **1925** Leica — 35 mm becomes the standard small format; **1935** anti-reflection **lens coating** (Smakula, Zeiss; multi-coating / T* in the 1970s — cuts flare, lifts transmission & contrast); **1935** Kodachrome (Kodacolor color-negative film follows in **1942**) — practical color film; **1948** Land — **Polaroid** instant photography (print develops in the camera; folding **SX-70** SLR in **1972**); **1963** Instamatic.

• **1969–75** the **CCD** (Boyle & Smith) and the first digital-camera prototype (Sasson, Kodak, 1975); **1977** Konica C35 AF — first autofocus camera; **1985** Minolta Maxxum 7000 — **phase-detection autofocus** in the SLR; **1995** Canon — first optically **image-stabilized** lens (in-body sensor-shift IS later).

• **the image industry went digital before the camera did (software & scanning)**: even while film still ruled *capture*, the photo *industry* moved to digital **images** — **desktop publishing / digital prepress** (late 1980s) scanned film for page layout; **drum and film scanners**, plus Kodak **Photo CD** (1992), digitised negatives and slides; **Adobe Photoshop 1.0** (**1990**, Thomas & John Knoll) put the **digital darkroom** on a desktop; and **photojournalism** digitised early (Kodak DCS bodies + wire transmission) because filing a picture *fast* beat film quality. Digital images were edited, scanned, and transmitted years before most photographs were *captured* digitally.

• **1990s** first digital cameras (Kodak **DCS** 1991, Apple QuickTake 1994, Nikon **D1** 1999); **2000** first **camera phone**.

• **2012** ImageNet / AlexNet — the deep-learning revolution; **~2014** GANs, then **~2022** **diffusion** models / DALL·E — generative image AI.

• **branches and oddities** to weave in: the **flash**; **holography**; **Lippmann** integral imaging and Lippmann interference color photography; light-field cameras (**Lytro**); and the coining of the term **"computational photography"** (Steve Mann, mid-1990s; broadened by Marc Levoy in the mid-2000s).

• **Sidebar: Notable cameras**

• **Polaroid Land Camera (1948) / SX-70 (1972)** — instant photography: the print develops in your hand, no lab. Edwin Land made the photograph immediate decades before digital; the folding SX-70 SLR (self-developing print) is a landmark of optics and industrial design.

• **Kodak DCS (1991)** — the first commercial digital SLR (a Nikon body + a 1.3 MP Kodak digital back); digital photography begins as a professional tool.

• **Apple QuickTake 100 (1994)** — one of the first consumer digital cameras (0.3 MP).

• **Nikon D1 (1999)** — the first DSLR built as an integrated digital camera (not a film body retrofitted), priced to start displacing film in photojournalism.

• **first camera phone — Sharp J-SH04 (2000, Japan)** — the camera becomes something you always carry; the road to phones as the dominant cameras.

• **Casio EX-F1 (2008)** — an early consumer **high-speed / burst** camera (60 fps stills, up to 1200 fps video) — a hint of the "shoot many frames" computational era.

• **Lytro (2012)** — the first consumer **light-field** camera (refocus after capture) — computational photography as a product category.

• computational photography spans a spectrum from **objective** to **subjective**:

• **objective** — physics and measurement: radiometry, optics, sensor noise, geometry, reconstruction error. There *is* a right answer — a deblurred image is closer to or further from the ground truth; SNR is a number; a calibration is right or wrong.

• **subjective** — taste and intent: what makes a "good" photograph, composition, a pleasing color grade, an artistic look. No single correct answer.

• **in between: perceptual** — how an image actually *looks* to a human. The surprise is that this middle ground is **fairly objective and quite well modelled mathematically**, along several dimensions: the **contrast sensitivity function**, color appearance, lightness/brightness, visual **masking**, perceived sharpness, and image-quality metrics (SSIM, VDP). "How it looks" can be measured and *predicted*, even though it is not pure physics — which is exactly what lets us **engineer for the eye** (tone mapping, chroma subsampling, perceptual metrics, where to spend bits or smoothing).

• this **objective → perceptual → subjective** spread is a recurring lens for the whole book: it helps to know *which kind of problem* you are solving before reaching for a tool.

• **two ideals of the photograph.** One tradition treats the photograph as a **window onto reality**: the ideal image is a *perfect reproduction* of the scene, so that viewing the picture is, perceptually, the same as having been there. **Gabriel Lippmann** pursued exactly this — his **interference colour photography** (Nobel Prize, 1908) recorded the full spectrum at each point, and his **integral photography** (1908) recorded directional rays (the ancestor of the light field) — both attempts to capture *everything* the eye could have received. **Helmholtz**, by contrast, spent his *Handbook of Physiological Optics* cataloguing how far the eye is from a clean instrument, which already hints that "reproduce the light exactly" and "reproduce the *experience*" are different goals.

• **but the medium is not transparent — it has hard limits.** A photograph is, physically, a **bounded, flat, low-contrast token** of an unbounded 3-D world:

• **dynamic range** — the scene spans far more contrast (sun to shadow, ~$10^4$–$10^8$:1) than paper or a screen can show (~$10^2$:1); something must give (→ [[Tone mapping]], [[HDR merging]]).

• **flatness** — a 2-D projection discards depth, parallax, and the directional structure of light (→ [[Light fields]], [[3D and depth]]).

• **limited field of view** — a frame is a narrow window cut from a $360°$ surround (→ panoramas, [[Many images and photo collections]]).

• (and more: fixed focus/aperture, a single instant, three colour channels for an infinite spectrum.)

• **the limits are also opportunities — they are the medium's *language*.** This is the key reframing (and the heart of [[#Objective, perceptual, and subjective|the perceptual/subjective view]], developed in **Durand 2002**): a constraint is not only a problem to overcome but an **artistic affordance**.

• **flatness as composition.** By collapsing 3-D onto a plane, the photograph lets the artist **put into relation objects that are far apart in the scene** — align a foreground branch with a distant peak, frame a doorway around a subject, build graphic patterns out of unrelated depths. Flattening is *why* composition (the 2-D arrangement of the picture) is a craft at all.

• **limited contrast as relation.** Because the medium can't preserve the scene's enormous luminance ratios, it **brings parts of the scene into a shared tonal world** — a face in shadow and a bright sky can be made to coexist on the page, relating luminances that the eye would never have reconciled at once. Tone mapping is, in this light, not just damage control but a *compositional* act.

• same for the frame (exclusion as emphasis), the single instant (the "decisive moment"), and shallow depth of field (isolation by blur).

• **takeaway / why it matters for this book.** Computational photography is exactly the toolkit for *negotiating* this tension: it pushes toward the **window** ideal when we want fidelity (HDR, wide gamut, stitched panoramas, light fields — chasing Lippmann's dream with code), and it leans into the **medium's** affordances when we want expression (tone curves, framing, synthetic depth of field). Knowing which ideal you are serving — faithful reproduction or expressive depiction — tells you which tools to reach for.

• the field is thrilling precisely because it sits at the **crossroads of an unusual number of disciplines** — you rarely get to touch this many at once:

• **algorithms** · **AI / machine learning** · **physics** (optics, light transport, radiometry) · **chemistry** (film, sensors, dyes) · **electrical engineering** (sensors, circuits, the ISP) · **mechanical engineering** (lenses, shutters, stabilization, camera bodies) · **computer graphics** · **computer vision** · **human perception** · **art** (the craft of photography) · **human–computer interaction (HCI)** · **performance engineering** (real-time, on-device, Halide) · **ethics**

• and they show up *together*: a single feature — a phone's **night mode**, say — leans on physics (photons), EE (the sensor), algorithms (align + merge frames), AI (learned denoising), perception (tone mapping), performance engineering (finish in a second on a phone), and ethics (how much to "beautify") **all at once**. That breadth is the fun of it.

• the cameras-and-pictures framing is the way *in*, but the machinery — form an image from measurements, model light/optics/sensors, invert with computation — is the same wherever images (or signals) are made.

• **the ordinary camera at extraordinary jobs** (the picture becomes an *input to a decision*, not the product): **robotics & self-driving** (perceive, localize, navigate — same alignment/depth/motion estimation), **surveillance/security**, **industrial inspection / machine vision** (defects on a production line), agriculture, drones, document scanning, biometrics.

• **more advanced optics**: **astronomy** (frame stacking = night mode at scale; adaptive optics; synthetic aperture — Event Horizon Telescope's Earth-sized virtual lens), **microscopy** (light-sheet / confocal / computational; super-resolution past the diffraction limit). Same game: co-design measurement + reconstruction.

• **other EM wavelengths**: infrared (thermal, night vision), ultraviolet, **X-ray → CT** (3D from many 1D projections — pure inverse problem), radio (radio astronomy, **SAR**).

• **medical imaging** is this discipline under another name: CT, **MRI**, **PET**, ultrasound — all regularized inverse problems recovering an image from indirect measurements.

• **non-imaging signals**: the toolkit (sampling, filtering, Fourier, denoising, priors, inverse problems) is really a **signal** toolkit. **Audio** is the obvious neighbor; example — Paul Lamere's **Infinite Jukebox** (find near-identical beats, jump between them → a song plays forever) is **Video Textures** (Schödl et al. 2000) transplanted from frames to sound.

• **computational imaging hiding in plain sight — the optical mouse**: an **optical mouse is a tiny visual sensor** (as the name says) — a **low-resolution camera** (a few tens of pixels square) plus an **image-processing chip** that tracks motion by **correlating successive frames** of the surface under it (the same cross-correlation / sub-pixel-shift estimation that powers alignment and optical flow), thousands of times a second. A computer-vision system shrunk into a $5 part you already own — computational imaging hiding in plain sight.

• **why imaging keeps reappearing — we really only measure electrons and photons**: a surprising amount of "non-visual" sensing is **visual under the hood**, because the two physical quantities we transduce well are **electrons** (charge, current, voltage) and **photons** (light). Electrons are the workhorse — almost every sensor ultimately produces a voltage — but **photons convert into electrons almost for free** via the **photoelectric effect**: a photon frees (or ejects) an electron, the insight that won **Einstein** his Nobel and exactly how a photosite works. So a huge range of measurements is staged as *make the quantity emit or modulate light, then count the photons as electrons*: fiber-optic and spectroscopic sensors, fluorescence assays, **scintillators** that turn X-rays / gamma into visible flashes for a camera, optical readout of pressure / strain / temperature, even reading bits off a disc. Photonic measurement recurs because the **photon ⇄ electron** conversion is cheap, fast, and linear — which is why this book's sensor, a photon counter, is a template far beyond photography.

• takeaway: you're not learning photography tricks but a **way of thinking about measured signals**, illustrated by photographs.

• **the audience**: a **CS undergrad** (or equivalent) — new to imaging, but comfortable thinking computationally. We lead **intuition-first** — the idea and a picture, then the math.

• **math — the small kit** (refreshed in [[Appendices#Refreshers|the Refreshers appendix]], so a rusty reader can catch up): **multivariable calculus** — derivatives and integrals, the **gradient** (and the chain rule; a light touch of **Jacobian / Hessian**), because a pixel is an integral of light and almost every recovery is a **gradient-driven optimization**; and **basic linear algebra** — vectors, matrices, dot products, and a little **eigen / SVD**, because *images are vectors and our operations are linear maps*. A little **probability** (noise) and a bit of **machine learning** round it out. Anything heavier lives in the Refreshers, not the main text.

• **programming — basic is enough, and the rest is learned as you need it**: you should be comfortable with **basic programming** — variables, loops, arrays and indexing, functions — ideally in **Python** (the book also ships **C++**; see *[[#What language for computational photography?]]*). **No advanced programming is required**: you do **not** need to arrive knowing NumPy, PyTorch, OpenCV, GPU/parallel programming, or performance engineering. Those are introduced **where they're needed** and can be **picked up as you go** — the worth-knowing libraries and the language trade-offs are surveyed in the next section, and performance optimization is its own (optional) part. The book deliberately has you implement core algorithms **from scratch**, which keeps the programming elementary and lets the *ideas* carry the weight.

• the book is **language-agnostic in its ideas** but ships in two editions — a **Python** one and a **C++** one (never one book showing both) — because the two languages serve different moments: Python to *think and prototype*, C++ to *ship and run fast*. The concepts are identical; pick the edition that matches your goal.

• **why choose which**: **Python** for **deep learning, prototyping, and convenience**; **C++** for **performance**. The gap is not small. When you write the **low-level code yourself** (e.g. your own convolution rather than a library call), the **speed differential** is dramatic: in Frédo's experience, roughly **1000× from Python to optimized C++**, and **at least another ~10× from C++ to Halide**. So the language choice is also a performance choice — see [[Performance engineering and Halide]] and [[Appendices#Programming: Python, C++, and PyTorch]].

• **Python** — the right default for *learning, prototyping, and research*:

• **pros**: **readable** — NumPy array code reads almost like the math (`out = a + 0.5*b`); **fast to write and explore**, with interactive notebooks and instant feedback; a vast **ecosystem** (NumPy, SciPy, Pillow / OpenCV, scikit-image, Matplotlib, and the whole deep-learning stack — PyTorch, JAX); automatic memory; the lingua franca of research and ML.

• **cons**: **slow at the pixel level** — a naive per-pixel Python loop is orders of magnitude slower than C, so you must **vectorize** (push the loop down into NumPy's C core) or it crawls; the **GIL** limits real CPU threading; packaging / deployment is awkward; dynamic typing lets type bugs hide until run time; little control over memory layout or exact performance.

• bottom line: the right default for **learning, prototyping, and research** — and, through NumPy / PyTorch, fast *enough* for most of what we do here.

• **C++** — the right default for *performance and production*:

• **pros**: **fast and predictable** — close to the metal, with control over memory layout, cache behaviour, and SIMD; what real **camera ISPs, real-time pipelines, and shipped products** are written in; static types catch errors at compile time; deploys to phones and embedded hardware.

• **cons**: **verbose and slower to write**; **manual memory management**, a rich source of bugs (out-of-bounds, leaks, use-after-free); a heavier **build / toolchain** story; far less interactive (edit–compile–run, not edit–run). The productivity tax is real.

• bottom line: the right default for **performance and production** — once the algorithm is settled and it must run fast, on device, at scale.

• **Halide** — a **domain-specific language** (embedded in C++ / Python) for high-performance image processing: you write the **algorithm** (what each pixel computes) **separately** from the **schedule** (how to tile, vectorize, parallelize, and fuse it across the memory hierarchy).

• **pros**: near-hand-tuned speed **without** hand-writing the tangled, hardware-specific loops; the *same* algorithm retargets to CPU / GPU / DSP by swapping the schedule; the basis of real camera stacks.

• **cons**: another language and mental model to learn, best reserved for the **hot paths** you have already identified — premature scheduling is its own trap. Covered properly in [[Performance engineering and Halide]].

• **MATLAB (historical note)**: for years *the* platform for this work — a strong numerical/linear-algebra library and an IDE that handled **images natively** (an image is a matrix; `imread`/`imshow`/slicing; the Image Processing Toolbox); much of the literature was prototyped in it. **Largely displaced by Python** (NumPy fills the same array niche, free) and especially **PyTorch** (autodiff + GPU + deep learning, which MATLAB never matched). Worth knowing because older papers/code are in it, and its array-thinking carries over to NumPy.

• **Libraries (implement-from-scratch, then build on)**: the book has you **implement core algorithms from scratch** to understand them (the `imageops` reference), but to *build* / dive into advanced projects, stand on mature libraries — **Python**: Pillow (PIL), NumPy/SciPy, OpenCV, scikit-image, PyTorch (also a GPU-array + autodiff engine); **C++**: OpenCV (de-facto standard), Eigen (linear algebra), LibRaw (raw decode), Halide (hot paths).

• **side bar — Photoshop 1.0, in 128,000 lines (a language lesson from 1990)**: Adobe (via the **Computer History Museum**, 2013) released the source of **Photoshop 1.0.1** — **≈128,000 lines across 179 files**, written for the Macintosh by **Thomas Knoll** (its sole engineer for v1 — a computer-vision PhD student whose 1987 image-display hack became the app). The language split *is* this section's lesson, 35 years early: **≈75 % Pascal** (the productive high-level language) and **≈15 % hand-written 68000 assembly** for the speed-critical inner loops — *most of it readable, the hot pixels hand-tuned* — exactly the Python-vs-C++/Halide trade-off (a high-level language for the bulk, machine code where it counts; what **Halide** now automates). The code is famously *mostly uncommented but well-structured*. It's now studied as a cultural artifact: the **"Das digitale Bild"** project (DFG / LMU Munich) runs an ongoing **Critical-Code-Studies** *"reading the source code of Photoshop"* — code as a text to be close-read. **For scale:** today's Photoshop is reportedly **millions** of lines of C++ (Frédo's estimate ~**10 M**); **Adobe Camera Raw (ACR)** and **Lightroom** share the same Camera Raw engine, and a typical camera **ISP** is a large proprietary C/C++ codebase — exact modern sizes mostly **not public** [⚠️ confirm the ~10 M / current ACR/Lightroom/ISP figures]. (→ the hand-tuned-inner-loop angle returns in [[Performance engineering and Halide]].)

• **Vibe coding (LLM-assisted)**:

• writing image code with an **LLM** ("vibe coding") is now part of the toolkit: great for **boilerplate, I/O, visualization, and explaining errors**, and a quick way to draft a function or a test.

• **but** image code is **numerically and perceptually subtle** — "looks plausible" is not "correct" — so keep the model on a **short leash** for filtering, resampling, color, and antialiasing, and always **verify against a hand-checkable input** (a constant, an impulse, a half-plane) before trusting it. Treat generated image code as *guilty until tested* (see [[Developing, Testing and Debugging]]). AI coding sometimes **fumbles the trivial** even when it nails the hard parts — see the cross-fade anecdote in [[Basic image processing and ISP#Vibe coding: writing image code with an LLM]].

• Although details can matter and can be subtle.

• In this chapter we give the basics and will worry about details later.

• a tiny image is literally a grid of pixels, each a square holding a few numbers (R, G, B) — show a 32×32 image with each pixel rendered as a square + a zoom to the array of numbers [pixel-grid figure]

• an image is stored as a **1-D block** of numbers; the *packing* (which index runs fastest) is a choice — interleaved **HWC** (NumPy default) vs planar **CHW** (ML), and the C++ vs Python conventions differ. Here we keep it light (just enough to address a pixel — our C++ uses `c·W·H + y·W + x`) and treat memory layout properly in [[Image representation]].

•  for now just think low value dark, high value bright,

• and one channel for each primary color (R, G, B)

• (we defer **gamma / encoding** to [[Light and physics]] and [[Image representation]]; here, just "bigger number = brighter")

• generate images: constant color, vertical gradient from black to white, Dirac, box, sine wave.

• Finger exercise: checkerboard. Line

• more advanced: mandelbrot? Sierpinsky?

• add or subtract constant, multiply by scalar (~brightening),

• non-linear gamma-ish curve for contrast (what’s a good curve ,

• finger exercise: convert to BW. Negative. Fake color scheme. Sepia (more red and green)

• translation, mirror, rotate by 180 degrees. Rotate by 90 degrees.

• Edge effects for rotate 90 degrees (return black for now?).

• Implement with loop over output, not input. Doesn’t matter yet, will matter later.

• blur

•  exercise: center-surround, finite difference gradient

• add

• subtract

• multiply? Graduated neutral density? Compositing/masking?

• look apple Photo Booth

• half mirror

• averages of humans? Provide pre-aligned images or use one of the celeb datasets

• three models: **ray** (geometry), **wave** (color, diffraction), **particle/photon** (counting → noise) — use whichever fits the question [wave/ray/particle figure]

• side bar — **wave–particle duality, made visible with a liquid analog**: light is *both* wave and particle, which is famously hard to picture. A startling **hydrodynamic analog** makes the marriage tangible: a tiny oil droplet **bouncing** on a vibrating fluid bath becomes a **"walker,"** carried across the surface by its own **pilot wave**, and these walkers reproduce quantum-like behaviour (single- and double-slit statistics, quantized orbits, tunnelling). They are *not* a theory of quantum mechanics, but they are a gorgeous classical picture of a particle that is inseparable from its wave. [embed video — Couder & Fort and **John Bush's** (MIT) walking-droplet experiments → [[Video Resources]]] (revisited at diffraction, below)

• wavelength, EM map, spectra

• **Newton's prism experiment** (the historical hook): a prism splits white light into the **spectrum** — white light is a *mixture* of wavelengths, and the *experimentum crucis* (a second prism) shows those pure colors don't split further. [classic diagram]

• a spectrum = power vs wavelength; like an **infinity of channels**, of which a camera (and the eye) keep only three → **metamerism** (Color technology)

• a camera sees nothing that hasn't bounced off something

• **reflection** (angle out = angle in); **refraction** (light slows in glass/water → *refractive index*; **Snell's law** n₁ sinθ₁ = n₂ sinθ₂); **diffraction** (→ Wave effects section below)

• marching-band (wavefront) intuition; fuller lens geometry deferred to Single lenses

• **interactive — a wave simulation of refraction** (`fig-refraction-wave-sim`): the same 2-D FDTD wave solver as the diffraction figure, but with a slower medium (higher index) filling the lower half of the box. An angled plane wave crosses the interface and its wavefronts **bend toward the normal** and **crowd together** (λ→λ/n) — the wave speed drops, so Snell's law becomes a *consequence* of the wave picture (the wave-level version of the marching-band intuition and Fermat's equal-path view in [[#Single lenses: refraction and Snell's law]]). Sliders for the index n₂ and the incidence angle, with an on-figure Snell check. Companion to `fig-diffraction-wave-sim`.

• **light–matter interaction is largely *resonance***: much of what light does to matter — the **index of refraction** itself, and **absorption** (hence material color) — comes from the **resonances** of atoms, molecules, and bound electrons. Light's oscillating field drives those bound charges, which resonate at characteristic frequencies (the **Lorentz-oscillator** picture); **near a resonance** the material **absorbs** strongly, and the **index of refraction varies with wavelength** (**dispersion**) according to how far each wavelength sits from the nearest resonance. This is why different materials transmit/absorb different colors, and why $n(\lambda)$ gives **prisms, rainbows, and chromatic aberration**. Flag forward: dispersion **shapes much of lens design** — achromatic/apochromatic doublets (crown+flint), the fight against chromatic aberration in fast lenses → [[Optics, lenses, and aberration correction]]. **Interactive** `fig-lorentz-resonance`: a driven bound electron (mass on a spring) with the absorption peak at ω₀ and the n(ω) normal→anomalous dispersion curve; sliders for driving frequency and damping.

• side bar: **polarization** — light is a *transverse* wave (field ⊥ propagation); **polarization** is the **orientation of that oscillation**. Natural light is **unpolarized** (orientation a rapidly varying mix, *no single direction*); reflection partly polarizes it → a polarizing filter cuts reflections / deepens the sky. Two everyday encounters: (a) **good sunglasses are polarizers** that selectively kill glare — *why* lives in the reflection/BRDF material (specular reflection off dielectrics is partially polarized, maximally at **Brewster's angle**) → [[#The BRDF and the look of materials]], the polarizer filter in [[#Lens filters: polarizers, ND, and graduated ND]], and reflection / specular removal in [[Single-image computational photography]]; (b) **LCD displays emit polarized light** — the liquid-crystal shutter forms an image by **rotating polarization between two crossed polarizers**, which is why a polarizer (or polarized sunglasses) makes some screens go dark at certain angles.

• **why some screens go dark through sunglasses and some don't.** Polarized sunglasses pass only **vertically** polarized light (so they can block the horizontally-polarized glare off roads and water). An LCD emits **linearly polarized** light fixed by its **exit polarizer**; whether the screen survives your sunglasses depends on *that polarizer's orientation*: if it emits **vertical** light it passes straight through, if **horizontal** the glasses **black it out**, and at **45°** it dims. Manufacturers choose the axis — laptops and monitors are usually safe in landscape, but many phones (and especially **car dashboards / GPS**, ATMs, and aircraft seat-back screens) are built with a diagonal or horizontal axis, so they wash out or go black when you tilt your head or rotate the phone 90°. The honest fix is to put a **quarter-wave retarder** *outside* the exit polarizer so the emitted light is **circularly** (not linearly) polarized — then it looks the same through any linear polarizer at any rotation; phones that stay bright through sunglasses generally do this.

• side bar: **why the sky is blue and the sunset orange** — **Rayleigh scattering**: air molecules scatter short (blue) wavelengths far more than long (red) ones (∝ 1/λ⁴). At midday the sunlight takes a *short* path through the atmosphere and the blue it sheds scatters into the sky in every direction → the sky reads **blue**; at sunset the light skims a *long* path, its blue is scattered away before it reaches us, and mostly **orange–red** survives → the sun reddens. (Larger droplets scatter all wavelengths about equally — **Mie** scattering → white haze and clouds.) [atmospheric scattering figure] [Minnaert]

• side bar: **rainbows** (spoiler: it's a caustic) — and a desktop demo. A **single spherical raindrop** bends sunlight three times: **refract** in at the front, **reflect** off the back, **refract** out again; because $n$ depends on wavelength (**dispersion**, above) each color exits at a slightly different angle. Across a whole sky of drops those exit rays **pile up** into a bright **caustic** at the **rainbow angle** — about **42° (red)** down to **~40° (violet)** measured from the **antisolar point** (the shadow of your head) — and *that* accumulation is the rainbow. A **glass sphere** reproduces it on a tabletop: it **stands in for a giant water droplet**, and a bright lamp throws its dispersed caustic onto a card (`fig-rainbow-glass-sphere`).

• **the color order, and the second bow**: the **primary** bow has **one** internal reflection — **red on the outside, violet inside**. A fainter **secondary** bow (~51°) comes from **two** internal reflections, with the colors **reversed**; between them sits **Alexander's dark band** (named for Alexander of Aphrodisias, ~200 AD), darker because one-reflection light emerges only at **≤42°** and two-reflection light only at **≥51°**, so the 42–51° gap gets almost no rainbow light (and the sky *inside* the primary is correspondingly brighter). Same geometry, one extra bounce. Real example: a **double bow over Loch Ness** (`fig-rainbow-loch-ness`, photo) showing both bows and the dark band.

• **dispersion alone is NOT a rainbow**: a prism only *fans* colours; the rainbow's brightness and fixed angle are a **caustic**. Vary the impact parameter and the deviation $D(b)$ of the **one-internal-reflection** ray bottoms out at a **minimum** (~138° → bow at 42°) where rays pile up. Crucially, a ray with **only two refractions and no internal reflection makes no bow** — its $D(b)$ is **monotonic**, no minimum, no pile-up. Dispersion only splits the already-bunched light into the band. Interactive Descartes diagram `fig-rainbow-droplet` (water drop; drag the impact parameter, choose the number of internal reflections, and tick **"zoom near the caustic"** to re-range both the slider and the plot tightly around the bow; the deviation-vs-$b$ plot marks the minimum, and a side panel plots the **ray density** — an integral over impact parameter — whose **spike** at the stationary deviation *is* the bow. Dispersion is always on, so three colour bands separate near the caustic).

• **it's an angle, not an object**: the bow is centered on the antisolar point, so it can't be reached and *everyone sees their own* — it is a **caustic of accumulated rays at a particular angle**, not a thing sitting in space (the same "it's a direction" lesson as vanishing points). [Minnaert] [`fig-rainbow-glass-sphere`, real demo photo; `fig-rainbow-droplet`, interactive]

• **scattering & absorption** in media (fog, haze, underwater): single vs. multiple scattering → de-weathering / dehazing; the same physics as the blue sky, at close range

• a **red apple is not red in the dark**: color = **illumination × reflectance**

• reflectance ρ(λ) is the surface's intrinsic, fixed signature; illumination S(λ) is whatever light is present; their per-wavelength product C(λ) reaches the eye

• this multiplicative entanglement is exactly what **white balance** must later undo

• scope caveat: the clean product assumes direction-independent reflectance (matte case); the general directional case is the **BRDF**, next

• side bar: **albedo can appear > 100% (fluorescence cheats)** — real (non-fluorescent) reflectance is **≤ 1 per wavelength** (a surface can't return more light than it receives at that wavelength). But a **fluorescent** material **absorbs** short-wavelength (often **ultraviolet**) light and **re-emits** it at **longer** wavelengths, so in the re-emitted band the measured "reflectance" **exceeds 1** — apparent albedo tops 100%. Examples: **optical brighteners** in laundry detergent and paper ("whiter than white"), **highlighter** ink, **fluorescent safety vests**. The trick is **moving energy between wavelengths**, not creating it.

• the **BRDF** (bidirectional reflectance distribution function): a surface re-emits incoming light into a *distribution* of outgoing directions — radiance toward q = f_r(p→q; n, λ) · incoming; *bidirectional* = depends on both the incoming and outgoing directions. [BRDF diagram]

• **diffuse (Lambertian) vs specular (shiny)**: a matte surface scatters into a broad lobe and looks equally bright from any view (the light it *receives* falls off as cos θ); a shiny one concentrates near the mirror direction, giving a view-dependent highlight; most materials are a blend. [Lambertian sphere vs shiny sphere]

• **side bar — where a material's colour actually comes from.** "Colour" is not one mechanism; the same red can be made several physically different ways, and they behave differently (some shift with viewing angle, some don't), which is why this belongs with the BRDF:

• **absorption (pigments & dyes)** — the everyday case: molecules/electrons absorb certain wavelengths (the **resonance** picture from *Light and physics*) and what's left is the colour. Paint, ink, skin, leaves (chlorophyll), most natural surfaces. **Angle-independent** and the basis of the *illumination × reflectance* model above.

• **structural colour (interference & diffraction)** — colour from **micro-geometry**, not pigment: **thin-film interference** (soap bubbles, oil slicks, anodized metal, the blue of a Morpho butterfly, peacock feathers), **diffraction gratings** (CD/DVD, holographic foil, some beetles), and **photonic crystals** (opal). The hallmark is **iridescence** — the hue **shifts with viewing/lighting angle** — because it comes from path-length differences, so it is strongly **directional** and a genuinely *spectral* BRDF effect, not a single albedo.

• **scattering** — wavelength-dependent **Rayleigh/Mie/Tyndall** scattering makes colour without any pigment: the **blue sky** and blue eyes, the **Tyndall blue** of some bird feathers and milk, the red of a sunset (cross-ref the Rayleigh sidebar in *Reflection, refraction*).

• **metallic reflection** — in metals the **free electrons** set the reflectance; most metals are grey/white (aluminium, silver), but **gold** and **copper** absorb in the blue and so reflect warm — a coloured *specular* with a tinted highlight (unlike dielectrics, whose highlight stays the light's colour). Matters for shading models (metal vs dielectric is the key axis in PBR).

• **emission & fluorescence** — the surface **makes** light: incandescence/LEDs (emit), and **fluorescence/phosphorescence** which absorb short λ and re-emit longer (the albedo->100% and "fluorescence breaks the BRDF" sidebars).

• takeaway: pigment colour is a per-wavelength **albedo** (the simple model), but interference, scattering, and metallic colour are **angle- and spectrum-dependent**, which is precisely why a faithful material model is a *spectral, directional* BRDF rather than a single RGB reflectance. [PBR: Pharr, Jakob & Humphreys]

• **light–matter interaction also polarizes the light**: reflection (especially the **specular** component) partially **polarizes** what bounces off a surface — maximally near **Brewster's angle** — so a full account of the BRDF is *polarization-dependent* (a **polarization BRDF / pBRDF**). This is exactly what a **polarizing filter** exploits to cut glare and reflections (→ Photography 101 — Lens filters) and what **shape-from-polarization** and reflection-removal methods use (→ Advanced). [cross-ref the polarization sidebar in *Reflection*]

• **side bar — why good sunglasses are polarizers.** Glare off roads, water, snow, and car hoods is light that reflected off a roughly **horizontal** surface, so (near Brewster's angle) it is strongly **horizontally polarized**. Quality sunglasses are **linear polarizers oriented to pass vertical and block horizontal** light, so they kill that glare specifically — not just dim everything like a plain neutral-density tint. That's the difference between "good" (polarized) and cheap (just dark) sunglasses: you see *into* the water and the wet road instead of at the sheen. The same physics is the camera's circular/linear polarizer, and the same selective extinction is shape-from-polarization's signal. (One catch: polarized glasses can make some **LCD screens / phone displays** go dark at certain angles, because those screens emit polarized light.)

• **try it yourself (how to test your sunglasses).** Hold the glasses up to a glary reflection — a car hood, a puddle, a window, a lake — and **rotate them ~90°**. If the glare **darkens in one orientation and comes back in the other**, they are genuine **polarizers**; if nothing changes, they are just tinted. (That is exactly what `fig-polarizer-sunglasses` shows. A second quick test: look at an LCD screen and tilt your head — polarized lenses go dark at some angles.)

• hint of **global illumination** (light bounces many times between surfaces) — and it is not just a rendering nicety, it is something the photographer actively **exploits** and occasionally **fights**:

• **handy — reflectors and bounce flash**: because light keeps bouncing, you can shape it by adding bounce surfaces. A **reflector** (a white/silver/gold panel) catches the key light and **bounces fill** into the shadows; a **bounce flash** fires at a ceiling or wall so the *whole room* becomes a big, soft secondary source instead of a hard on-axis point. Both are global illumination put to work — softening contrast and wrapping light around the subject (→ Photography 101 — fill & bounce flash; [[#Human factors and the art of photography]]; computational bounce flash in Computational illumination). [photo: someone holding a reflector to bounce fill onto a subject — `fig-reflector-fill`, **shoot our own** unless a CC image turns up]

• **a challenge — unwanted bounced color**: the same bounced light can **hurt** you, because it carries the *color* of whatever it bounced off (color bleeding — the very thing the `fig-global-illumination` figure shows). The classic case is shooting at a **pool**: the strong **blue** light bouncing up off the water casts a cold tint on faces that makes decent **skin tones** genuinely hard to hold (a green lawn or a red wall does the same in its own hue). Knowing the bounce is colored is half the fix — block it, add neutral fill, or correct it later.

• **side bar — fluorescence breaks the BRDF.** A BRDF is **per-wavelength**: same wavelength in, same wavelength out, just redistributed in direction. **Fluorescence** violates this — it **absorbs short-λ (often UV) light and re-emits at *longer* wavelengths** (the **Stokes shift**), moving energy *between* wavelengths. So a fluorescent material is not described by a scalar BRDF per λ but by a **reradiation (bispectral) matrix** — input wavelength → output wavelength. (This is also the cheat behind apparent albedo > 100% — cross-ref, don't re-derive, the albedo sidebar in [[#The color of objects: illumination times reflectance]].) Everywhere once you look: **optical brighteners**, **highlighter** ink, **minerals under UV**, **GFP** in bio-imaging, **banknote security inks** — and it is the working signal in fluorescence **microscopy** and **forensics**.

• **side bar — subsurface scattering (why skin isn't plastic).** A BRDF assumes light leaves at the *same point* it entered. In **translucent** materials light instead **enters, scatters around beneath the surface, and exits somewhere else**, so a point-wise BRDF is **insufficient** — you need the **BSSRDF** (bidirectional *surface-scattering* RDF), which couples an entry point to a separate exit point. Examples: **skin**, **marble**, **milk**, **wax**, **leaves**. This is what gives skin its soft, "lit-from-within" glow and rounded shadow terminator; render skin with a pure BRDF and it looks **plastic**. (Cross-ref translucency / global illumination in rendering.)

• 'white' light is not one thing — the same scene at noon vs sunset shifts color

• **blackbody** & **color temperature** (low K warm/orange, high K cool/blue); **D65**, **tungsten** (warm), **fluorescent** (spiky → poor color rendition)

• insist again: the stimulus is the **spectral product** (illuminant × reflectance) → motivates white balance

• **radiance** L: power per unit area per unit solid angle carried *along a ray* — ultimately what a pixel measures; **conserved along a ray** in vacuum (a wall doesn't get dimmer as you step back, only smaller)

• **irradiance** E: power per unit area *arriving on a surface* (the sensor) — the integral of incoming radiance over the collecting solid angle, which grows with **aperture area**

• the **cos θ** in E = ∫ L cos θ dω is **projected area**: a beam (cylinder of light) of width w striking a surface at angle θ spreads over a larger footprint w/cos θ, so the same power is thinned → E ∝ cos θ (Lambert's cosine law). [cosine / projected-area figure]

• **sidebar — seasons**: the same cosine explains seasons — the Earth's 23.5° tilt makes summer sun strike North America near-vertically (concentrated) while winter sun arrives at a grazing angle (spread out, ~half the energy per area). [seasons figure]

• **units — radiometric vs photometric** (worth pinning down): the quantities above are **radiometric** (raw energy, in **watts**). Weight them by the eye's spectral sensitivity — the **luminous-efficiency curve $V(\lambda)$** — and you get the **photometric** quantities the photographic world actually quotes:

• radiant flux **W** ↔ luminous flux **lumen (lm)**

• irradiance **W/m²** ↔ **illuminance lux** (lm/m²) — light *arriving on* a surface (an incident-meter reading)

• radiance **W/m²/sr** ↔ **luminance cd/m²** (the "**nit**") — light *along a ray* / leaving a surface; this is the perceptual "brightness" of a patch, and what displays, HDR, and the photopic/scotopic scale (Perception) are all quoted in

• intensity **W/sr** ↔ luminous intensity **candela (cd)** — the SI **base** unit (≈ one candle)

• so **cd/m² = lm/(m²·sr)**; for scale, a monitor is ~100–300 cd/m², an HDR display ~1000+, paper in sunlight ~10⁴, and the photopic/scotopic ranges (Perception) live on this same axis

• **exposure** H = irradiance × time → ties straight back to aperture (D/f)² and shutter t (Exposure settings)

• **falloff**: a *point* source dims as **1/r²** (inverse-square), but scene **radiance is distance-invariant** — contrast the two carefully, it confuses people

• this radiometry underlies **HDR** (recovering scene radiance) and **computational illumination**

• (the off-axis **cos⁴** edge falloff / natural vignetting is deferred to **Sensors**, where the pixel irradiance integral and a finite aperture make it meaningful)

• light is a **wave**: at edges and small apertures it **diffracts** — bends and spreads instead of travelling in straight rays

• consequence for imaging: a **finite aperture cannot focus to a perfect point**; even an ideal lens images a point as a small blur, the **Airy disk**, whose size grows with wavelength × f-number

• the **diffraction limit**: the finest resolvable detail is set by wavelength and aperture (Rayleigh ≈ 1.22 λ/D); a **larger** aperture diffracts **less**

• so **stopping down too far softens** the image — aberrations dominate wide open, diffraction dominates stopped down → a sharpness **sweet spot** (developed with MTF in the Optics part)

• **coherent vs incoherent**: we mostly assume **incoherent** imaging (intensities add, stay positive, linear); coherent / wave optics returns for advanced optics (metalenses, lensless, holography)

• **a live demo — the wave equation, solved**: diffraction through a slit is shown as an actual 2-D wave simulation (FDTD), wavefronts bending and spreading past the opening, with the time-averaged intensity pattern alongside — and a companion widget where a *smaller aperture visibly spreads more* (θ ≈ λ/D). [`fig-diffraction-wave-sim` (animated), `fig-diffraction-aperture-size`]

• **the audio parallel** (use it — sound is the everyday 1-D wave): every wave effect here has a **sound** counterpart, and most people have *heard* them. Diffraction — **bass bends around a doorway** (long λ) while treble stays directional, exactly as a small aperture spreads light; plus **interference** and **beats**, **standing waves** in a pipe or room, and the **Doppler** shift. The governing math is shared (a wave equation, Fourier content, λ/D spreading), so acoustics is a reliable intuition pump for the optics — and the same FDTD solver in `fig-diffraction-wave-sim` *is* an acoustic simulation with light's numbers.

• side bar — **diffraction and duality, revisited with walkers**: diffraction is the *wave* story (a slit spreads light by θ ≈ λ/D); the *photon* story is that **individual** photons still arrive one at a time, yet pile up into the same fringe pattern. The **walking-droplet** analog (see [[#Light: rays, waves, and the spectrum]]) dramatizes the marriage: a single bouncing drop passing a slit is steered by its own **pilot wave** into a **diffraction-like** distribution — so you can literally *watch* a "particle" guided by a wave. [reuse the Couder/Bush walker video → [[Video Resources]]]

• pinhole,

• show camera obscure example (some artist)

• **geometry of pinhole imaging**: each scene point's chief ray passes through the pinhole onto the sensor. The physical sensor sits *behind* the pinhole, so the image is **inverted** (upside-down and left-right flipped).

• we instead use a **virtual image plane the same distance in front** of the pinhole — *not physical*, but it keeps the image **right-side up**, so we adopt it throughout (consistent with the camera looking down **+z** → [[Conventions, Notation & Style#Geometry & coordinate systems]]).

• the pinhole→sensor distance is the **focal length f**: a short f gives a **wide** field of view, a long f a **narrow** (telephoto) one — FOV = 2·arctan(sensor / 2f).

• **side bar — DIY pinhole.** A pinhole needs no lens, so it is the easiest real camera to build: punch a clean tiny hole in foil and tape it over the lens mount of a camera body, or over one end of a lightproof box (a shoebox / oatmeal tin) with film or paper at the other — a working **camera obscura**. Smaller holes are sharper but dimmer (diffraction sets the floor), so exposures run long.

• **side bar — the pinhole bag** (credit the idea to **Freeman, Torralba & Isola**'s online vision book, their Figure 5.5, at <https://visionbook.mit.edu/imaging.html>): pull a bag with a single small pinhole over your head, let your eyes dark-adapt, and you see the **upside-down image of the bright world outside projected on the inside of the bag** — pinhole projection made personal, and an unforgettable demo that the inverted image is real.

• exercise: make your own, do in a room

• mention animal eyes

• linear perspective when camera is at origin and along z axis

• **perspective projection equation**: a scene point (X, Y, Z) projects to screen coordinates **x' = f·X/Z** and **y' = f·Y/Z** — each screen coordinate is the world coordinate **divided by the depth Z and multiplied by the focal length f** (the "divide by z" that makes distant things small). [shown both as 2D similar-triangles and in a 3D view]

• 💡 **Big lesson — with the camera at the origin looking down +z, perspective is *mostly a division by depth*.** In that canonical frame the whole of linear (pinhole) perspective is just $x' = f\,X/Z,\ y' = f\,Y/Z$: divide each world coordinate by the depth $Z$, then scale by $f$. Everything else a camera does — sitting somewhere else, pointing some other way, landing on pixels — is bolted on *around* this one step. The **divide-by-depth is the whole of perspective**; the rest is bookkeeping. → register [[Big Lessons]]

• **homogeneous coordinates**: write a 2D point as (x, y, 1) up to scale, so projection, **translation**, and camera motion all become **matrix multiplies**

• **translation becomes a matrix, too.** A shift by $(t_x, t_y)$ is *not* linear on $(x, y)$, but on the homogeneous triple it is an ordinary matrix multiply: $\begin{bmatrix}1&0&t_x\\0&1&t_y\\0&0&1\end{bmatrix}\begin{bmatrix}x\\y\\1\end{bmatrix} = \begin{bmatrix}x+t_x\\y+t_y\\1\end{bmatrix}$, and in 3D the $4\times4$ analogue $\begin{bmatrix}I_3&\mathbf{t}\\\mathbf{0}&1\end{bmatrix}$ translates $(X, Y, Z, 1)$. This is *the* reason graphics and vision live in homogeneous coordinates — translation, rotation, projection, and the perspective divide all compose into one chain of matrix multiplies. [translation-matrix in homogeneous coordinates]

• so far the camera has sat at the **origin looking down +z**, where perspective is the clean divide-by-depth above. But a real camera is **somewhere else, pointing some other way** — so before we can use that canonical projection we must re-express the scene **in the camera's own frame**. This is the step that turns "an arbitrary camera" into "the camera at the origin."

• 💡 **moving the camera = applying the *inverse* transform to the world.** Sliding the camera right by $d$ is indistinguishable from sliding the whole **world left** by $d$; rotating the camera one way is rotating the world the other. So we never move the camera *inside* the projection — we move the **world into the camera's frame** by the inverse rigid transform, then project with the fixed origin/+z pinhole. (This is also why "camera-to-world" and "world-to-camera" matrices are inverses — the convention trap below.)

• that world→camera step is a **rotation R then a translation t**, written as one $4\times4$ matrix in homogeneous coordinates, $\begin{bmatrix}R&\mathbf{t}\\\mathbf{0}&1\end{bmatrix}$, carrying a world point $(X, Y, Z, 1)$ into the camera frame — where the canonical divide-by-depth then applies. Translating the camera is changing $\mathbf{t}$; aiming it is changing $R$.

• 💡 **Big lesson — perspective for an *arbitrary* camera is one $4\times4$ matrix followed by a divide by the last coordinate.** Pose the camera (world→camera $4\times4$), project, and map to pixels — every stage is a matrix on the homogeneous point, so the *whole* geometric pipeline composes into **one matrix multiply**, and the only nonlinearity — the perspective divide — *is* the final "divide through by the last coordinate." The arbitrary-camera case is the origin case (divide by depth) wrapped in two more matrices; homogeneous coordinates are exactly what make that wrapping elegant. → register [[Big Lessons]]

• moving camera, matrices, all that good stuff

• **camera intrinsics K** — give them a real treatment, not a bullet: reason first in **normalized image coordinates** (project onto the z=1 plane, independent of sensor size), then map to **pixels** with a second, **affine** matrix carrying the focal lengths fx, fy and the **principal point** (the arbitrary image centre). So the full projection **decomposes** as *(pixel affine) · (project at z=1) · [R|t]* = K·[R|t]. This split is also why **cropping an image = using a longer focal length** — you change only the pixel mapping, not the projection. [projection-matrix decomposition figure; crop = focal-length figure]

• 💡 **Big lesson — cropping a photo *is* changing the focal length.** From the projection geometry, cropping touches only the last stage, the pixel mapping $K$: keep a smaller patch of the normalized image and rescale it to fill the frame — exactly what enlarging $f_x, f_y$ does. A 2× crop and a 2× zoom produce the *same* picture (the crop just has fewer pixels). Scaled up to whole sensors, this same fact is the **crop factor** that relates sensor sizes. → register [[Big Lessons]]

• **camera extrinsics [R|t]** — where intrinsics are *inside* the camera, extrinsics are the camera's **pose in the world**: a **rigid 6-DOF** world→camera transform, a rotation **R** (3×3, 3 DOF — orientation) plus a translation **t** (3 DOF — position). Together they place the world into the camera's frame *before* projection, so the **full projection is intrinsics · extrinsics = K·[R|t]** (the [R|t] moves the world into camera coordinates; K projects and maps to pixels). "Moving the camera" is just changing R and t.

• **the convention trap — cam2world vs. world2cam** (the one place to slow down): the matrix [R|t] above maps **world → camera**; its inverse maps **camera → world**. Worse, packages disagree on whether **t is the camera's position** (its center C in world coordinates) or its **negation** in the projection (t = −R·C). Mixing the two silently flips signs / mirrors a reconstruction — always pin down *which direction* a stored pose maps and *what t means* before trusting it.

• this is exactly the machinery that **multiple-view geometry** and **panorama homographies** rely on — relating two cameras is composing their extrinsics; a pure rotation between views collapses [R|t] into a **homography** (→ [[Multiple view geometry]]; Pano & homographies).

• vanishing points

• **what projection preserves vs destroys**: straight **lines → lines** and incidence survive; **angles and lengths do not** (parallels converge to vanishing points, a circle can become an ellipse) — the core fact behind perspective

• Photography, focal length, field of view, various trigomometric ideas, magnification. Difference focal length vs distance and how that changes the background

• the same focal-length-vs-distance distinction governs **portraiture**: facial proportions (and even perceived personality) are set by **camera-to-subject distance**, which is *why* portrait lenses (~85–135 mm) flatter — they let you back up (Perona 2007; full treatment in [[#Lenses and focal length]]).

• sensor size and impact on perspective. Maybe too late?

• sidebar: the (weird) meaning of **"35 mm"** = a **24×36 mm** image frame. The "35 mm" is the *width of the film stock* (including the sprocket-hole margins), not the image; Barnack ran 35 mm cine film sideways, giving a 36 mm × 24 mm frame. Today this size is called **"full frame"**, and smaller sensors are quoted by their **crop factor** relative to it (APS-C ≈ 1.5×, Micro 4/3 ≈ 2×, phone sensors much smaller).

• **depth vs. ray length**: a depth sensor's "depth" is the **z coordinate**, *not* the ray length ‖·‖ to the point — keep them distinct; **unproject** (pixel + depth → 3D point) inverts the projection (RealSense & friends do this live). Depth sensors exist (→ ToF / structured light / stereo). [depth-vs-ray-length / unproject figure]

• a real lens is a piece of glass bounded by **two curved surfaces** (two interfaces)

• light bends at each interface by **refraction** — Snell's law: n₁ sin θ₁ = n₂ sin θ₂

• shown explicitly on a **thick lens** (the two interfaces drawn far apart): the angle of incidence and the angle of refraction at *each* surface, with the indices n₁ (air) and n₂ (glass) marked. [thick-lens Snell figure]

• a curved interface bends a bundle of rays; the two surfaces of a single lens together steer parallel light toward a **focal point**

• **a second view — Fermat / equal optical path length**: a lens focuses because **every** ray from the object to the focus has the *same optical path length* — the longer geometric detour through the thick centre is exactly compensated by the slower speed (higher index) of glass there, so all paths arrive **in phase** and interfere constructively. The ray (Snell) picture and this wave (Fermat) picture are two takes on the one fact. [equal-path-length figure]

• 🖱️ **Interactive (web edition):** a live **2-D wave simulation of a thick lens** — a point source on the left radiates circular waves that cross a biconvex glass slab (slower inside, higher index n); because the glass is thickest on the axis it retards the centre of each wavefront most, turning the diverging wave into a **converging** one that collapses to an **image point**. This is the Fermat/wave picture made literal: focusing emerges from the wave equation, not from a ray rule. Sliders for the surface curvature R, the index n, and the source distance u, with the focus tracking the **lensmaker's equation** $1/f=(n-1)\,2/R$ and the **imaging equation** $1/v=1/f-1/u$ live on the figure. The lens sits in an opaque mount (aperture stop) so all light is forced through the glass; absorbing (sponge) borders, so nothing reflects off the window edges. Companion to `fig-refraction-wave-sim` (single interface) and `fig-diffraction-wave-sim`. [`fig-thick-lens-wave-sim`; interactive, FDTD]

• **lensmaker's equation** ties focal length to the two radii of curvature and the glass index: 1/f = (n−1)(1/R₁ − 1/R₂)

• this already assumes small angles (paraxial); at large angles the rays do **not** meet at one point → **aberrations** (forward ref: Optics part, where compound lenses correct them)

• eyes have optics too (cornea + lens are the two main refracting elements)

• the **thin lens** as a *simplification / linearization* of the single lens: collapse the two interfaces into one plane, assume small angles and negligible thickness → all bending happens once, at the lens plane (forward ref: Optics part for the accurate thick / compound model)

• precisely, **paraxial = first-order optics**: replace **sin θ ≈ θ** (keep only the linear term). Keeping the *next* term — sin θ ≈ θ − θ³/6 (**third-order optics**) — is exactly what produces the **Seidel aberrations**, so "linearization" and "no aberrations" are the *same* approximation (forward ref: Optics part)

• three-ray construction (parallel→focus, through-centre, focus→parallel) locates the image

• **focus equation** 1/f = 1/u + 1/v (object distance u, image distance v); an object at infinity focuses at f; **focusing** = changing v (move the lens or the sensor) to image nearer/farther objects

• **conjugates**: each object distance pairs with one image distance — object at ∞ → image at f; far → just beyond f; close → far behind; at the focal point F → rays exit parallel (image at ∞). [conjugates figure]

• **symmetry & reversibility**: the focus equation $1/f = 1/s_o + 1/s_i$ is **symmetric** in object and image distance, so object and image are **conjugate** — **swap them and the rays retrace the same path** (the reversibility of light paths). Object and image planes are **conjugate planes**, and the magnification follows directly ($m = -s_i/s_o$, so swapping inverts it). This object↔image **reciprocity** recurs throughout optics (e.g. why a lens images either way, and why entrance/exit pupils are conjugates).

• **magnification** m = −v/u (the image is inverted)

• **lensmaker's equation** (thin-lens form) 1/f = (n−1)(1/R₁ − 1/R₂) — connects f to the glass shape (from the single-lens section)

• **f-number** N = f/D sets the aperture (→ exposure, → blur)

• (depth of field gets its **own section**, next)

• **defocus / circle of confusion**: a point off the focus plane images to a point that is not on the sensor, so the aperture cone meets the sensor as a blur disk — the **circle of confusion**. A point still counts as sharp while its disk stays within an acceptable c. [CoC figure]

• the blur disk grows with the **aperture** (wider → bigger disk) and with the **object's distance from the focus plane** (zero exactly at focus); focal length and focus distance enter too. [defocus-vs-parameters figure]

• **depth of field** = the *range* of object distances whose blur disk stays within c; the **near** and **far** limits are where the blur just reaches c. [DoF near/far figure — each cone carried out to the CoC at the sensor]

• **deriving the limits**: on the image side a near-limit object images *behind* the sensor and a far-limit object *in front* of it; **similar triangles** relate the aperture D, that convergence distance, and the circle of confusion c — and together with the conjugate relations (1/f = 1/s + 1/v) they fix the near (front) and far (back) object-distance limits. [near-limit and far-limit similar-triangle figures]

• **hyperfocal distance** H = f²/(N·c): focus there and everything from **H/2 to infinity** is acceptably sharp — the focus setting that maximizes DoF (focusing at ∞ instead wastes the H/2…H zone). [hyperfocal figure]

• DoF **grows** with a smaller aperture (larger N), a shorter focal length, and a greater focus distance — these are the paraxial (linearized) formulas; the accurate model is in the Optics part. [DoF-vs-parameters figure]

• **the surprising invariance** (the lecture's flagged "important conclusion"): for a **fixed framing** (same subject size) and a **fixed f-number**, the depth of field in **object space** is essentially **independent of focal length** — switching 28 mm → 100 mm forces you to step back, and the larger physical aperture cancels the longer focal length. So "long lenses have shallow DoF" is really about **magnification**, not focal length per se. (The *background* still looks more blurred with the long lens — that's magnification of the out-of-focus disk, not a DoF change.) [same-framing/same-N → same DoF figure]

• ⚠️ **but the invariance is only *first-order* — the background gives the focal length away.** "Same framing + same f-number → same DoF" is the **small-defocus (paraxial)** result: it holds for the **near/far acceptable-sharpness limits straddling the subject**, but **not** for how hard a *distant* background is thrown out of focus. **Derivation** (thin lens; blur-disk diameter on the sensor): a point at distance $s_b$, with the lens focused at $s_f$, images to a circle of confusion $c \approx \dfrac{f^2}{N}\,\dfrac{|s_b-s_f|}{s_b\,s_f}$. **Fix the framing** — subject magnification $m=f/s_f$ held constant, so $s_f=f/m$ — and **fix $N$**, with the background a gap $\Delta=s_b-s_f$ behind the subject:

$$c(f)=\frac{m^2\Delta/N}{\,1+ m\Delta/f\,}\ \xrightarrow[f\to\infty]{}\ \frac{m^2\Delta}{N}.$$

The subject-zone DoF is ~independent of $f$ (the leading $f$ cancels — the **first-order invariance**), but the **background blur grows monotonically with focal length**, saturating at $m^2\Delta/N$. Two shots matched in framing *and* aperture render the **subject** identically, yet a **200 mm melts a far background far more** than a 35 mm: the correction term is order $m\Delta/f$ — negligible for a background near the focus plane (invariance looks exact), but dominant once the background is many focus-distances away. This is exactly why portrait shooters reach for long lenses for "background separation" even though the DoF on the *face* is unchanged. **Full derivation + plots** of $c$ vs. background distance for 35 / 85 / 200 mm (fixed framing & $N$) — the curves coincide near the focus plane and fan apart far from it. [`fig-dof-background-blur`]

• **object-space picture & a caveat**: the simplest view is a **double cone** in front of the lens whose waist sits on the focus plane — it gives the blur for **one** scene point. But **defocus is *not* a spatially-invariant convolution**: the circle of confusion depends on each point's **scene depth** (not its pixel location), and at depth discontinuities **occlusion** matters — which is why naive "just blur the image" fakes look wrong, and why light-field / depth-aware methods (Advanced) do better. [object-space double cone; depth-dependent-blur + occlusion figure]

• **sensor size scales DoF**: shrink the sensor (keeping FOV) and both the circle of confusion and the aperture shrink, so DoF grows **linearly** with sensor size — why phone cameras have huge DoF (and resort to *computational* background blur), and why **macro is easier on small sensors** (a scaled-down camera photographs a scaled-down world — modulo diffraction)

• 🖱️ **Interactive (web edition):** a live **macro depth-of-field simulator** — a to-scale thin-lens diagram, a lens-supersampled 3D photo (real bokeh), and a circle-of-confusion-vs-distance plot all driven by one shared optics model (focal length, f-number, sensor, focus distance, subject–background gap). A 3D bug (beetle / bee / ladybug) over a flower makes the macro lesson vivid: focus lands on the front of the face and the rest of the body, antennae, and background all fall out of focus. [fig-depth-of-field-sim; interactive] *(bug & flower 3D models generated with Meshy AI — acknowledge in the caption)*

• (placed here, just before **sensors**, because it pays off as the framing for what a pixel actually measures)

• the **plenoptic function** describes *all the light filling space*: the radiance of every **ray**, parameterized by its 3D position, 2D direction, wavelength, and time — a 7-dimensional function [Adelson & Bergen]

• **any** imaging operation is the **measurement of some portion** of this function; the eye and the camera each sample a slice of it

• a conventional camera records only a **2D projection**: each pixel integrates all the rays reaching it, **discarding their direction** — information that richer ray-based / multi-aperture devices (Advanced part) can recover

• a recurring unifying view: imaging = sampling the plenoptic function

• **the chapter's thesis in one equation — a pixel value is an integral of the [[#The plenoptic function|plenoptic function]]** against a sampling kernel: each photosite *counts photons*, so it reports the **incoming radiance integrated over four physical dimensions**, weighted by the pixel's response. Schematically,

• pixel value ≈ ∫_time ∫_λ ∫_aperture ∫_(pixel area) L(x, y, θ, φ, λ, t) · r(λ) · (geometry) dA dω dλ dt

• i.e. **measurement = integrate the plenoptic function against the pixel's sampling kernel** — and the kernel's *support* in each dimension is exactly a photographic knob. [fig-pixel-integral: thin lens as an ellipse (aperture solid angle), the sensor, one finite pixel; the integral over aperture + pixel area]

• **each integration dimension ↔ a photographic parameter ↔ a downstream effect** (every later sensing topic is "what happens when you change the support of one axis"):

• **sensor footprint (x, y area)** — the pixel integrates radiance over its **finite area** on the sensor, which sets **resolution** and makes a sample a *box-filtered* average rather than a point sample → **sampling & anti-aliasing** (a bigger footprint blurs but suppresses aliasing; pre-filter vs. point-sample lives here).

• **aperture (solid angle of admitted rays)** — the pixel integrates over the **cone of directions** the lens admits, set by the **f-number** N = f/D. Wider aperture = more light *and* a wider integration cone = a shallower **depth of field** (out-of-focus points spread their energy across many pixels → the circle of confusion). The light-vs-blur trade is this single axis. (→ [[#Depth of field]])

• **exposure time** — the pixel integrates over the **shutter interval** t, which sets exposure (H = E·t) and, when the scene or camera moves during t, produces **motion blur** (the width of the temporal kernel). Light-vs-sharpness, the temporal sibling of aperture. (→ Motion / video)

• **wavelength × spectral sensitivity** — the pixel integrates radiance against its **spectral response** r(λ), collapsing a whole spectrum to one number (exactly like a cone, → [[#Color]]). A **color-filter array** gives each photosite a different r(λ) (R/G/B), so color is *spatially multiplexed* and must be reconstructed → **demosaicking** (Bayer).

• **why this framing pays off**: the rest of the sensing story is just *which kernel, how wide, and what it loses* — antialiasing widens the spatial kernel, depth of field is the aperture kernel, motion blur is the time kernel, demosaicking inverts the wavelength multiplexing. Each is also a **source of noise**: integrating fewer photons (small footprint, short time, narrow aperture) means a noisier estimate (→ Noise, next). Capture is **linear** in radiance, which is why this integral is well-defined and why **raw** values *mean* something physical.

• the **digital-sensor revolution came in two steps**: first **electronic** — turning light into an *analog electronic signal* (the photosite + readout; the era of the CCD and analog electronic / video imaging) — and only **then digital** — putting an **analog-to-digital converter (ADC)** on the path so that signal becomes **numbers** we can compute on. Computational photography lives on that second step; flag the distinction up front.

• a **sensor** is a 2-D array of **photosites** (pixels), each a potential well that **counts photons** → charge → voltage → a digital number; counting photons is what makes capture ~**linear** in radiance (keep this short — the deep sensor zoo is Advanced)

• a **microlens** over each photosite gathers light onto its active area (**fill factor** < 100% because of wiring), and a **color filter** sits under it → the **Bayer** mosaic (forward-ref: Demosaicking). [sensor cross-section figure]

• the scene-radiance → pixel-value mapping is the **camera response curve**: **raw** is ~linear in radiance, while the rendered (JPEG) image applies a non-linear response (≈ gamma + a tone curve — see Linear-vs-gamma in Color technology); **recovering** that response is exactly what HDR merging needs

• **CCD vs CMOS**: a **CCD** shuttles charge across the chip in a "bucket brigade" to **one** amplifier at the edge (clean, uniform, but slow and power-hungry, and it reads the whole frame at once → naturally **global** shutter); **CMOS** puts an amplifier (and now the ADC + logic) **at every pixel** and reads in place, **row by row** (fast, cheap, low-power — now utterly dominant). [CCD bucket-brigade vs CMOS read-in-place schematic]

• what CMOS's per-pixel, row-by-row readout **unlocks** (and costs): on-chip ADC, region-of-interest and high-frame-rate readout, on-sensor HDR and PDAF and binning — but, because rows are read at different times, an **electronic rolling shutter** with its motion artifacts (below).

• the pixel value is an **integral over four dimensions** (aperture × pixel area × wavelength × **time**) — the [[#Image measurements as integrals]] chapter owns the first three; **this is the time one**. Over the **exposure time** Δt the well **accumulates** photon arrivals: value ∝ ∫ irradiance dt. [photons-accumulating-in-a-well figure]

• **exposure = irradiance × time** (the other half of the exposure triangle with aperture; ISO is gain *after*). **Reciprocity**: ½ the light for 2× the time gives the same exposure — until it doesn't (sensor reciprocity holds far better than film's reciprocity *failure*).

• the time integral is exactly **motion blur**: anything that moves during Δt is averaged along its path → a long exposure smears motion (light trails, silky water), a short one freezes it. The deliberate use and the *removal* of this integral are [[Motion blur and temporal sampling]] and deblurring.

• **what ends the integral is the shutter** — which is the rest of this chapter.

• **mechanical shutter** — a physical curtain (or leaf) opens then closes to time the exposure. A focal-plane shutter is **two curtains**: at short times the second starts closing before the first fully opens, so a **slit** sweeps the frame (already a kind of rolling exposure, in hardware). A **leaf shutter** (in the lens) opens from the centre → exposes the whole frame at once (global) and **syncs flash at any speed**.

• **electronic shutter** — no moving parts: the sensor **resets** each photosite to start its integration and **reads it out** to end it. Silent, vibration-free, and capable of very short or very high-frame-rate exposures — but on CMOS it is usually **rolling** (below). Many cameras use an **electronic first curtain** (electronic start, mechanical end) to cut shutter shock while keeping a clean finish.

• **global shutter**: every pixel integrates over the **same** time window (all start and stop together), then the frame is read out. No motion distortion. Native to CCD and to special global-shutter CMOS; costs transistors/area per pixel.

• **rolling shutter**: rows are exposed and **read out one after another**, so each row's time window is **shifted** slightly later than the one above. Cheap and standard on CMOS — but rows see the scene at different instants. [row-vs-time diagram: global = a rectangle in (row, time); rolling = a slanted parallelogram]

• the **artifacts** (all from "different rows, different times"): **skew** (a fast pan or a moving car leans over), **wobble / "jello"** (handheld video shimmer), the **spinning-propeller / guitar-string** surreal shapes, and **partial exposure with flash** (a brief flash lights only the rows open at that instant → a **bright band**, which is why flash sync speed exists). [📺 [rolling shutter explained](https://www.youtube.com/watch?v=dNVtMmLlnoE) — Smarter Every Day; `fig-rolling-shutter` interactive: a vertical rod / spinning blade under global vs rolling readout]

• **lighting & flicker banding**: mains-powered and many **LED/PWM** lights **flicker** (100/120 Hz, or fast PWM dimming); under a rolling shutter, rows captured at different phases of that flicker get different brightness → horizontal **banding** (and colour banding if RGB phosphors lag). Fixes: match the exposure to the flicker period, **anti-flicker** modes that time the readout, or a global shutter. Ties to video [[Video stabilization and rolling-shutter correction]].

• **correcting rolling shutter** computationally — estimate the per-row time offset and the camera/scene motion, then warp each row back to a common instant → [[Video stabilization and rolling-shutter correction]].

• **off-axis (cos⁴) falloff** still applies once there's a sensor + finite aperture: rays reaching the frame edge arrive obliquely → **cos⁴θ** dimming toward the corners (**natural vignetting**), distinct from optical vignetting (Optics).

• **dual conversion gain (DCG)** — a CMOS pixel can switch the **sense-node capacitance**, changing the conversion gain (µV per electron): **high-conversion-gain** mode (small capacitance → lower read noise, better low light / high ISO) vs **low-conversion-gain** mode (large capacitance → bigger full well, more highlight headroom). Two settings → two **native ISOs** (**dual native ISO**); reading/combining both extends dynamic range → [[Noise, signal-to-noise ratio and dynamic range]].

• **quad-Bayer (Tetracell / quad CFA)** — the CFA groups **2×2 same-color photosites** under one filter tile. Good light: the sensor **remosaics** back to full-resolution Bayer; low light: it **bins** each group into one larger effective pixel (less noise, ¼ resolution). How 48/50/108 MP phone sensors get clean low-light output and in-pixel staggered-exposure HDR. Variant: **3×3 Nonacell**. → [[Demosaicking]], [[HDR merging]].

• **dual-pixel and quad-pixel autofocus (on-chip lens, OCL)** — split a photosite into **2 (dual-pixel)** or **4 (quad-pixel, 2×2 OCL)** sub-photodiodes under **one microlens**; each sub-diode sees a different half/quadrant of the lens pupil → **phase-detection autofocus (PDAF) at every pixel** (a stereo baseline inside the pixel); sub-diodes are **summed** for the final image (no light loss). Sensor structure only — the optics/AF algorithm are in [[Focus, autofocus]].

• two eyes / two cameras → **disparity** (nearer points shift more)

• **random-dot stereograms** (Julesz): two random-dot images identical except for a shifted patch fuse into a floating surface — proof that **disparity alone yields depth**, with no recognizable objects or monocular cues at all ("depth without objects") [random-dot stereogram figure]

• simple parallel-camera geometry: depth ∝ baseline · focal length / disparity

• sidebar — **widening your baseline**: since disparity ∝ baseline, gadgets that optically push your eyes farther apart exaggerate depth (**hyperstereo**) — Helmholtz's **telestereoscope** (mirrors, 1857), military **optical rangefinders** & the **scissors periscope** (*Scherenfernrohr*), vs **hypostereo** (shrunk baseline) for macro 3D

• disparity map; rectification (a homography) when the cameras aren't parallel

• ties to panorama stitching; forward ref: multi-view stereo & depth (Advanced)

• the key fact, in one picture: a point in one view does **not** pin its match to a point in the other — it constrains it to a **line**. Given a pixel in the left image, its 3-D point lies somewhere along a single ray; that ray projects, in the right image, to a line — the **epipolar line**. So matching is a **1-D search along a line**, not a 2-D search (this is exactly why we *rectify* stereo pairs — to make those lines horizontal). [fig-epipolar-line]

• **derive the matrix straight from that picture** — ray → plane → line → algebra:

• **(1) the ray (depth ambiguity)**: a pixel back-projects to a ray; in **calibrated** coords ($\hat{x}=K^{-1}x$) the 3-D point is $X=\lambda\hat{x}$, depth $\lambda$ unknown.

• **(2) the epipolar plane**: that ray together with the **second camera centre** spans a plane (equivalently: the plane through both camera centres and $X$ — independent of $\lambda$).

• **(3) the epipolar line**: the plane meets the second image in a line, so the match $\hat{x}'$ must lie on it.

• **(4) write the plane in 3-D**: with relative pose $(R,t)$ ($X\mapsto RX+t$; the first centre maps to $t$ = the **baseline** vector, the ray direction maps to $R\hat{x}$), the three vectors $\hat{x}'$, $t$, $R\hat{x}$ are **coplanar** → vanishing triple product $\hat{x}'\cdot(t\times R\hat{x})=0$ → $\hat{x}'^{\mathsf T}[t]_\times R\,\hat{x}=0$ → the **essential matrix** $E=[t]_\times R$.

• **(5) → the fundamental matrix**: in raw pixel coords, $x'^{\mathsf T}Fx=0$ with $F=K'^{-\mathsf T}[t]_\times R\,K^{-1}=K'^{-\mathsf T}EK^{-1}$; $Fx$ is the epipolar line. The relative pose $(R,t)$ (translation only up to scale) is recoverable from $E$ — the seed of multi-view reconstruction.

• **forward-ref** (deferred to Advanced/3-D part): *estimating* F/E from noisy matches (the **8-point algorithm**, RANSAC), rectification, triangulation, and **structure-from-motion / multi-view stereo**.

• light, lens, pupil, retina,

• cones/rods, distribution, fovea — note there are **no S (blue) cones in the very centre of the fovea**, and far more L+M than S overall, so **blue is low-resolution and a bit out of focus** (chromatic aberration) — yet we perceive blue everywhere (the brain fills it in). Luminance ≈ the **L+M cone sum**, which is why **green carries most of the acuity** (and why Bayer uses 2× green).

• cells, ganglions, on-retina spatial processing, visual cortex, V1, higher.

• **the eye as a camera — and its refractive errors.** Like any lens system the eye can mis-focus: **myopia** (eyeball too long / cornea too curved → image forms *in front* of the retina), **hyperopia** (too short → *behind*), **astigmatism** (a non-spherical cornea → two focal lines, not one point), and **presbyopia** (the aging lens stiffens and loses **accommodation**). Each is corrected by a lens quoted in **diopters** (D = 1/focal-length in metres): a negative/diverging lens for myopia, positive/converging for hyperopia, a **cylindrical** lens (with an axis) for astigmatism. [`fig-refractive-errors`]

• **how the prescription is *measured* (the machines), and why it's just inverse imaging.** **(1) Retinoscopy** — shine a moving streak of light into the eye and watch the **retinal reflex** sweep across the pupil; its direction and speed reveal myopia vs hyperopia, and trial lenses are stacked until the reflex stops moving ("neutralization"). **(2) Autorefractor** — automates this objectively: it projects an **infrared** target onto the retina and analyses the **returning image** (its focus/size, or a Hartmann–Shack / Scheimpflug pattern) to solve for the corrective lens that makes the retinal image sharp — essentially a little camera running autofocus *on your eye*, in seconds. **(3) Phoropter** — the familiar "better one… or two?" wheel of trial lenses over an eye chart: a **subjective** refinement of the machine's objective estimate. **(4) Wavefront aberrometer (Shack–Hartmann)** — for the full optical picture: a **lenslet array** samples the wavefront leaving the eye and measures **higher-order aberrations** (coma, spherical, trefoil), not just sphere/cylinder — the same wavefront sensing used in **adaptive optics** and in planning **LASIK**. The throughline: measuring an eye is **inverse imaging** — find the optics that would make the retinal image (or the returning wavefront) ideal. (Cross-ref wavefront / Shack–Hartmann sensing and adaptive optics in [[Optics, lenses, and aberration correction]].)

• cones, projection from spectrum to 3 responses

• relate to measurement equation in image formation

• **the response is a matrix–vector product**: discretize the spectrum into N wavelength bins → a vector E ∈ ℝᴺ; stack the three cone sensitivities as the **rows of a 3×N matrix C** (one row per L/M/S); the cone responses are then **r = C·E**, i.e. each r_k = Σ_i c_k(λ_i)·E(λ_i). In the real world wavelength is **continuous**, so it's really **infinite-dimensional** — the sum becomes the integral r_k = ∫ c_k(λ)·E(λ) dλ. [cone-response matrix figure]

• insist a given cone does not make the difference between wavelength. Example of monochromatic stimuli

• point out they are non orthogonal and lots of stuff can’t be negative.

• This make linear algebra more messy

• 💡 **Big lesson:** **color is non-orthogonal *and* non-negative** — overlapping cone axes plus the no-negative-light constraint are what make color algebra tricky (analysis needs a dual basis with negative coordinates).

• side bar: **non-orthogonal bases & the dual basis** — the cone "axes" are **not orthogonal** (L and M overlap heavily), so the comfortable orthonormal intuition fails. Picture two basis vectors **c₁, c₂ both in the positive quadrant** of the (canonical) spectrum space, each standing for one cone. **Synthesis** is easy — any point is a combination a₁c₁ + a₂c₂ (a parallelogram). But **analysis** — recovering the coordinates a₁, a₂ — is **not** a plain projection onto c₁, c₂; you must project onto the **dual (reciprocal) basis** c₁\*, c₂\* (defined by cᵢ\*·cⱼ = δᵢⱼ). For a non-orthogonal positive basis the **dual vectors point partly into the negative quadrants** — they have **negative coordinates** in the original space. That is exactly why "stuff can't be negative" makes color algebra messy: the natural *analysis* vectors are negative. (Only for an **orthonormal** basis do a basis and its dual coincide — then analysis = synthesis = projection.) [non-orthogonal dual-basis figure]

• color blindness. Linear algebra perspective (projection loses info)

• we are all color blind

• side bar — **animal vision and the divergence of opsins**: "color" is just whatever set of **opsins** (photopigments) an animal carries, and these have **diverged enormously** across major animal groups — many insects are UV–blue–green trichromats, birds and reptiles are often **tetrachromats** (a fourth, UV cone), most mammals reverted to **dichromacy** (primate trichromacy is *re-evolved*, via a gene duplication), and the **mantis shrimp** carries ~12–16 photoreceptor classes. So human red–green color blindness is one point in a vast space — every species is "color blind" relative to some other. The opsin family tree and its divergence are traced in the "Evolution of Eyes" chapter of *Vision* (Cambridge); the schematic opsin gene tree (ancestral opsin → rod RH1 + cone classes, human S/M/L with the recent L/M duplication) is [fig-opsin-tree]. [→ Animal eyes]

• the **multistage color model** — the brain re-codes color in stages (simplified; see Reinhard et al., *Color Imaging* for the full version):

• **stage 1 — cones**: three heavily-overlapping LMS responses (above)

• **stage 2 — opponent recoding** in the retina / LGN: LMS → **light–dark, red–green, blue–yellow** (decorrelates the channels, matches the four unique hues; → next section)

• **stage 3 — appearance level**: a higher organization closer to **hue / saturation / brightness** (an HSV-like arrangement) where context, adaptation and memory act (→ color appearance models, Color technology)

• **focal colors & naming**: color *names* cluster around shared **focal colors** across languages (Berlin & Kay; the World Color Survey) — evidence the recoding is perceptual, not arbitrary

• experimental support: **psychophysics** (hue-cancellation — Hurvich & Jameson) and **physiology** (opponent cells — De Valois)

• the perennial muddle (RGB? RYB? CMY?) has *two separate* causes, neither of which is a fact about light:

• **dual-process / opponent coding** (next section): the visual system re-wires the three cone signals into **red–green, blue–yellow, light–dark**, so *red, green, blue, and yellow* all feel "primary" perceptually — there are **four** psychological unique hues, not three. This is why RYB feels natural to painters and why the cones' actual L/M/S don't match anyone's intuition of "primaries."

• the rest is **additive vs subtractive synthesis** (Color technology): **RGB** are the *additive* (light) primaries, **CMY** the *subtractive* (ink/filter) primaries; grade-school **RYB** is a folk version of subtractive mixing. So "primary" conflates a **perception** fact with a **technology** choice — different goals, different "primaries."

• source: lecture — the opponent rewiring "is one of the many causes of all the confusion about what's a primary color" (transcript 10/2); pair with [[Glossary]] entry for *primary*.

• two stages: trichromatic at the cones, then opponent in the retina / LGN — red–green, blue–yellow, light–dark

• explains afterimages and color naming; and why **luma/chroma** (YCbCr — luma Y′ on gamma-encoded RGB, see *Linear vs Gamma*) compresses well (see JPEG)

• we recenter the neural response around the current average brightness — spanning a huge range over *time*, not all at once

• mechanisms: neural, chemical (photopigment bleaching), and the pupil

• afterimages; the Spanish-Castle illusion

• **the operating regimes — photopic, mesopic, scotopic** (which receptors are working, and *how bright* that is, in **luminance** cd/m²):

• **photopic** (daylight, **cones**, full color, sharp foveal vision): roughly **> 3 cd/m²** — office light ~10²; an overcast sky ~10³–10⁴; a sunlit scene ~10⁴–10⁵

• **mesopic** (dusk / dim interior, **rods *and* cones together**): roughly **10⁻³ – 3 cd/m²** — colors desaturate and the **Purkinje shift** sets in (as rods take over, blues look relatively brighter and reds darker)

• **scotopic** (night, **rods only** — no color, poor acuity, and *off-fovea* since the fovea is all cones): below ~**10⁻³ cd/m²** — moonlit landscape ~10⁻²; starlight ~10⁻³–10⁻⁴; the **absolute threshold** of vision ~**10⁻⁶ cd/m²**

• big picture: the eye works across a staggering **~10⁻⁶ to ~10⁸ cd/m²** (≈ 14 log units), but only over a few log units **at once** — which is exactly why **adaptation** (above) and, in cameras, **HDR / tone mapping** exist. [fig-photopic-scotopic-range]

• the visual system reports a surface's intrinsic **lightness** (its reflectance / albedo), **not** the raw luminance reaching the eye — a white page in shadow still looks white, a black one in sunlight still looks black, even when the shadowed white sends *less* light to the eye than the sunlit black

• it **discounts the illuminant**: the brain factors the retinal image into **illumination × reflectance** (the same entanglement from Light and physics) and reports the *reflectance* — the property we actually care about

• this is an **ill-posed inversion** (one image, two unknowns), so the brain leans on cues and priors: **edges and context** tell it where the *illumination* changes versus where the *surface* changes

• the canonical demo: **Adelson's checker-shadow illusion** — two squares of identical luminance look very different because the brain infers the cast shadow and discounts it

• **Retinex** (Land & McCann) models the same idea computationally: treat slow spatial gradients as illumination and sharp edges as reflectance changes, then integrate the edges back to lightness

• ties to **intrinsic images** (split a photo into reflectance + shading) and to the darkroom craft of **dodging & burning** / tone mapping

• the **color analog** of lightness constancy: we perceive an object's intrinsic **surface color** (its reflectance) despite large changes in the **illuminant** — a white shirt reads white under noon daylight, a warm lamp, or blue shade, even though the spectra reaching the eye differ a lot

• again the brain **discounts the illuminant**: it estimates the light's color and divides it out so the surface reads consistently — the same ill-posed (**illuminant × reflectance**) inversion, solved with assumptions (e.g. the average scene is roughly gray, or the brightest patch is white)

• a key mechanism is **von Kries adaptation**: an *independent gain on each cone type* (L, M, S) rescales the response to the prevailing light — plus spatial / contextual cues; it is **not perfect** (some cast remains, and memory colors bias us)

• this is the **biological counterpart of white balance** (Color technology) — a camera's automatic white balance must do, by algorithm, what our visual system does for free

• classics: **Retinex** (Land & McCann); **Bayesian** color constancy (Freeman & Brainard); gray-world / bright-pixel heuristics

• the **"the dress"** demo (2015): the *same* image pixels look **blue-black** to some viewers and **white-gold** to others — because the brain must *guess the illuminant* (cool daylight vs warm indoor light) and divides it out differently. A perfect, viral illustration that color constancy is an **ambiguous inference**, not a measurement — the visual system commits to a prior and different people commit differently. Ties to white balance (a camera faces the exact same ill-posed guess).

• contrast is about *ratios*: 1→2 looks like 100→200 (discounts the multiplicative effect of light) — foreshadows gamma / log encoding

• Weber's law / just-noticeable difference

• simultaneous contrast; the Adelson checker-shadow illusion

• Visual Acuity

• Simultaneous contrast

• Mach bands; lateral inhibition (center-surround receptive fields)

• **CSF** — measured by **psychophysics** (show **sine-wave gratings**, ask "do you see a grating?"); the **contrast sensitivity function** is band-pass in spatial frequency. Crucially the **chromatic** CSFs (red–green, blue–yellow) cut off at **far lower** spatial frequency than the **luminance** CSF — we resolve **color** much more coarsely than **brightness**. [three-CSF figure]

• the dramatic demo: in Lab, **blur the chroma** channels and the image still looks sharp; blur the **luminance** and it falls apart — the empirical basis for **chroma subsampling** (JPEG / YCbCr) and a confirmation of the opponent model. [blur-chroma vs blur-luminance figure]

• the visual system trades some spatial resolution for **temporal** resolution: a **temporal CSF** band-passes flicker, with a **critical flicker-fusion** frequency (~50–60 Hz in bright light, lower when dim) above which a flickering source looks steady — why displays/film refresh fast enough to fuse, and why old CRTs / fluorescents could flicker in peripheral vision

• **apparent motion** (the phi phenomenon) + persistence of vision make a sequence of frames read as continuous movement (→ Motion/Video)

• **temporal adaptation** (light/dark adaptation over seconds–minutes; → Light adaptation) and the eye's own motion (saccades, microsaccades) shape what we see over time

• exploited by computation: **frame-rate** choices, flicker-free capture, and **video magnification** (amplifying tiny temporal changes — Motion/Video)

• attention; fixations; saccades; smooth pursuit

• the eye samples a scene through rapid saccades + a high-res fovea

• saliency (what draws the eye) is modeled computationally → see [[Human factors#Computational models of perception]] (the model and its uses) — the ML estimator is in [[Computational tools#Machine learning]]

• color reproduction kind of stuff

• blue shift

• colorfulness as a function of luminance (Hunt effect)

• hue shifts: Bezold–Brücke (with intensity), Abney effect (with added white)

• **the eye evolved as a sequence of small improvements**, and it recapitulates the optics of this book — from "no image" to a focused one. Following Nilsson's stages (Land & Nilsson, *Animal Eyes*):

• **a flat patch of photoreceptors** (an *eyespot*): just **photosites**, **no image formation** — it tells light from dark and, crudely, which side has *more* light (phototaxis), but cannot form a picture. This is the bare **sensor** with no optics.

• **a concavity / cup**: fold the patch into a **pit** and the rim's shading gives **directional** sensitivity — the deeper the cup, the better you can tell *where* light comes from.

• **a pinhole**: close the cup almost shut → a **pinhole eye**, a real (if dim) **image** with no lens at all (the **nautilus** still uses one) — exactly the camera obscura of [[#Pinhole image formation and linear perspective]].

• **a lens**: fill the aperture with a **refractive lens** (often a graded-index sphere) → a **bright, focused** image, independently evolved in fish, cephalopods, and vertebrates. Add a **cornea, iris, and accommodation** and you have the human eye.

• so the human eye is *one* solution; evolution reached the same goal by very different routes (below).

• **a tour of those other solutions** — a useful contrast to the human eye and to camera design:

• **compound eyes** (insects): many **ommatidia** → wide field of view and fast temporal response, but low spatial resolution

• **more cone types**: birds / reptiles are **tetrachromatic** (a 4th cone, into the UV); the **mantis shrimp** has ~12–16 photoreceptor classes (yet poor color *discrimination* — it seems to recognize rather than compare) — the **divergence of opsins** across animal groups is the color sidebar in [[#Color]]

• **tapetum lucidum** (cats, deer): a reflective layer behind the retina that boosts night vision (and causes eye-shine)

• **polarization vision** (cephalopods, many insects): they see light's **polarization** — a channel we're blind to

• **other eye designs**: **pinhole** (nautilus), **mirror eyes** (scallops — image formed by a concave mirror, not a lens), and the front-facing (**stereo**) vs side-facing (**field-of-view**) placement trade-off

• further reading: Land & Nilsson, *Animal Eyes*; the "Evolution of Eyes" chapter in *Vision* (Cambridge); a friendly overview at phos.co.uk, "The Evolution of Sight".

• analysis = *sensing* color (project a spectrum onto 3 cone / sensor responses)

• synthesis = *reproducing* color (add a few primaries)

• both lose information: the projection is non-orthogonal and light can't be negative — a tension that recurs (sensing, displays, white balance)

• the color-matching experiment: match a test light by mixing 3 primaries

• trichromatic theory (von Helmholtz 1859); tristimulus values

• metamers: different spectra that match — *why* color reproduction is possible at all

• CIE 1931: color-matching functions, the xy chromaticity diagram, the spectral locus, the white point

• negative matches → why some colors need "impossible" primaries (see Reproducing color)

• different challenges: quantization, noise

• same observation: ratios matter

• **side bar — why gamma ≈ 2.2? the CRT accident**: old **CRT** displays had a *natural* power-law response — screen luminance ∝ (grid voltage)^γ, γ ≈ 2.2–2.5 (electron-gun physics: beam current vs control-grid voltage is a power law). So a CRT **automatically decodes** a gamma-encoded signal: send V = L^(1/γ) and the tube returns L = V^γ (linear light), no decode hardware. **sRGB's ≈2.2 codifies it.** Bonus: that curve ≈ the *inverse of human lightness perception* (Weber–Fechner), so it also puts code levels where the eye is sensitive — **why we kept gamma after CRTs vanished** (LCD/OLED emulate the curve for compatibility). [CRT-response figure]

• **side bar — even sRGB isn't quite continuous**: the ubiquitous sRGB transfer curve is **piecewise** — a short **linear** toe near black, $V = 12.92\,L$ for $L \le 0.0031308$, then a **power** segment $V = 1.055\,L^{1/2.4} - 0.055$ above it — meant to join smoothly at the knee. But the standardized constants (the $12.92$ slope, the $0.0031308$ breakpoint, the $0.055$ offset, the $1/2.4$ exponent) are **rounded and not mutually consistent**, so the two pieces **don't exactly meet**: at the breakpoint the linear part gives $\approx 0.04045$ while the power part gives $\approx 0.04059$ — a tiny **discontinuity** ($\sim 10^{-4}$), on top of the slope kink that's there by design. So even the encoding *every* image uses is, strictly, **not continuous** at the join. The lesson: real transfer functions are messier than the clean "$\gamma \approx 2.2$" story — **check the actual curve** (and its inverse) before doing arithmetic or quantizing in that space. *(A "corrected" sRGB nudges the breakpoint/scale so the pieces meet exactly.)*

• **side bar — gamma in film**: the word **gamma** comes from photography. Film's response is the **characteristic (Hurter–Driffield) curve** — **density** vs **log exposure** — with a toe (shadows), straight middle, and shoulder (highlights). Film **gamma = the slope of the straight portion** (γ = ΔD/Δlog H) = the medium's **contrast**, set by development (push-processing raises γ). So *film* gamma is a contrast/rendering parameter while *display* gamma is an encoding power law — both log–log slopes, different jobs. End-to-end **system gamma** (capture × display) is tuned slightly >1 (~1.1–1.5) so images look right in a **dim viewing surround**. [film characteristic-curve figure]

• 💡 **Big lesson:** **additive vs multiplicative → choice of encoding.** Light *adds* (incoherent sources, blur) but surfaces and contrast *multiply*; **log** makes the multiplicative native, **linear** suits the additive (deconvolution), **gamma** is the compromise with least crazy behaviour near zero.

• **log is becoming the standard for *video***: high-end and now prosumer cameras capture in **log** profiles — **S-Log / S-Log3** (Sony), **Log-C** (ARRI), **V-Log** (Panasonic), **C-Log** (Canon), **RED Log3G10** — and the **ACES** pipeline standardises a log working space (ACEScc / ACEScct). Log packs a wide scene dynamic range into limited bits while keeping the grade malleable: a **stop** becomes a roughly constant code increment, so exposure and contrast moves are uniform. The footage looks flat and desaturated straight out of camera and is **color-graded** to a display encoding; the broadcast-HDR transfer functions **HLG** and **PQ** (SMPTE ST 2084) are the display-side cousins. So in video the trio increasingly resolves to **log for capture, gamma / PQ / HLG for display**. (→ HDR, tone mapping.)

• **luma vs luminance** (a recurring confusion): **luminance** Y is the *linear*, perceptual brightness — a weighted sum of **linear** RGB (CIE Y); **luma** Y′ is the *same-shaped* weighted sum computed on the **gamma-encoded** RGB (the coding convenience behind YUV / YCbCr / JPEG). They differ because the weighting and the gamma don't commute.

• **our convention**: use **Rec. 601 weights — Y = 0.299 R + 0.587 G + 0.114 B** — matching video / JPEG / YCbCr and our programming exercises; mention **Rec. 709** (0.2126 / 0.7152 / 0.0722) as the HDTV / linear-light alternative.

• **RGB and its flavours**: many RGB spaces differ only by their **primaries** (gamut) and **white point**, related by a single **3×3 matrix M** — sRGB / Rec. 709 (web, HDTV), **Adobe RGB** and **Display P3** (wider), **Rec. 2020** (UHD), **ProPhoto** (very wide, for editing). Converting between them (or to XYZ) is one matrix multiply *in linear light*.

• **opponent / luma-chroma spaces** (YCbCr, YUV, YCoCg): a fixed 3×3 map separating **luma** from two **chroma** axes — used for compression (chroma subsampling) and some editing.

• caveat: "linear" here means **a linear map between spaces**, not necessarily linear-*light* values — sRGB carries a gamma; XYZ is linear-light.

• **CIELAB** (and CIELUV): a deliberately **non-linear, perceptually-uniform** space — a cube-root compression of XYZ about a white point yields **L\*** (lightness) and **a\*, b\*** (opponent chroma), so **equal distances ≈ equal perceived differences**; color difference **ΔE = ‖ΔLab‖** (improved by ΔE2000)

• the motivating demo: sample a rainbow at **N colors equally spaced in sRGB vs in CIELAB** — the sRGB steps have wildly uneven perceived gaps (ΔE), the Lab steps are even [perceptual-uniformity figure]

• **why non-linear**: perception is roughly power-law (Weber–Fechner / Stevens), so a uniform space must warp XYZ — the same reason gamma encoding exists

• **friends**: HSV / HSL (handy for UI, *not* perceptually uniform), and modern uniform spaces — CAM02-UCS, **Oklab** — that fix CIELAB's known non-uniformities (the *Chong* perceptual color metric belongs here)

• intro: fundamental challenge of spectrum non-orthogonality.

• Give example of reproducing a monochromatic signal.

• Show 2D linear algebra equivalent problem.

• Say that the big extra challenge is that we can only have positive light.

• Non-orthogonality + restriction to positive -> impossible problems.

• **additive vs subtractive** color synthesis — cartoon: split the spectrum into 3 equal bands and pretend they are pure **B, G, R** [3-band synthesis figure]

• **additive** (lights): start from black and *add* bands → R+G+B = white; pairs give yellow / cyan / magenta

• **subtractive** (filters/inks): start from white and each filter *removes* a band — but this is really a wavelength-by-wavelength **multiplication** of the spectrum, so "subtractive" is **misleading**; C·M·Y → black

• concrete proof it's a multiplication: a **yellow** filter (blocks blue) stacked with a **cyan** filter (blocks red) passes only the overlapping green band → **yellow × cyan = green**, not a "sum" of the two [subtractive spectra-multiplication figure: T_Y(λ)·T_C(λ)]

• the **overlapping-disks** picture makes the same point at a glance: RGB lights overlapping toward white (additive) vs CMY inks overlapping toward black (subtractive) [overlapping RGB/CMY disks figure]

• real systems are often **hybrid** — some light *added* and some *removed* — e.g. an **LCD projector**: a white lamp, LCD panels that *subtract* (modulate) each channel, then the three channels *added* on the screen

• various technology: CRT, LCD, projectors (DMD, LCD, laser)

• film, printer, halftoning

• gamut, gamut mapping

• the problem: **the same numbers look different on different devices** — each camera / monitor / printer has its own primaries, white point, gamut and transfer curve

• the solution — **ICC color management**: every device carries a **profile** mapping its values to/from a device-independent **profile-connection space** (XYZ or Lab); image → PCS → target device makes color *portable*

• **rendering intents** for out-of-gamut color: **perceptual** (compress the whole gamut, keep relations), **relative colorimetric** (clip + white-point match), **saturation**, **absolute colorimetric**

• **chromatic adaptation** between white points via a **Bradford 3×3** transform (the engineering cousin of von Kries adaptation, Perception)

• in practice: profiles embedded in files (→ File formats), monitor calibration, soft-proofing; **sRGB** as the safe default when no profile is present

• four ways to turn a monochrome sensor into a color one — **multiplex** the three (or more) measurements across some axis:

• **in time**: shoot R, G, B frames sequentially through filters — Maxwell's first color photograph (1861), **Prokudin-Gorskii**, flatbed scanners, astronomy filter wheels. Cheap and full-resolution, but **fails on motion** (the classic color fringes on moving water).

• **in space (the dominant choice)**: a **color filter array (CFA)** — the **Bayer mosaic** (2 green : 1 red : 1 blue) — one color per photosite, then **demosaick** to fill the rest; the eye does the same with its interleaved cone mosaic. Variants: Fuji's **X-Trans** (a larger, less-periodic tile to fight moiré) and **complementary CMY/CYGM** CFAs (more light through, messier color). Needs an **optical anti-aliasing (low-pass) filter** to tame high-frequency color artifacts, and suffers some channel **crosstalk**.

• **beam-splitter (prism)**: a **dichroic prism** splits the rays onto **three sensors** — **3-CCD / 3-chip**, the classic **broadcast video camera** — wasting no photons at full per-channel resolution, but bulky, costly, and hard to align.

• **in depth (stacked)**: stack wavelength-selective layers so one location senses all three — **Foveon** (silicon absorbs longer wavelengths deeper; used in **Sigma** cameras), echoing **Kodachrome**'s stacked dye layers. Full color at every pixel (**no demosaicking**), but trickier color separation and more chroma noise.

• **spectral / Lippmann**: record the standing-wave **interference** of the full spectrum (Lippmann, 1908 Nobel; a precursor to holography) — true spectral capture, not just 3 numbers.

• **hybrid**: combine these (e.g. spatial + temporal in video).

• this is *analysis* (sensing color) — the mirror of color *synthesis* (reproduction, above); the **demosaicking** algorithm itself is deferred to Basic Image Processing (which is why this is just the *sensing* half)

• **trichromatic (perceptual) vs spectral (physical) capture**: almost all imaging systems aim to capture **perceptual, trichromatic color** — three numbers that reproduce what a *human* would see (RGB ≈ the cones), so metamers are, by design, indistinguishable. **Multispectral / hyperspectral** imaging instead samples the **physical spectrum** in many narrow bands (tens to hundreds), keeping distinctions the eye throws away — for remote sensing, agriculture, art conservation, and machine vision, where the *material*, not the *appearance*, is what matters (and metamers must be told apart). Lippmann (above) is the limiting case: the full continuous spectrum.

• **saturation**: scale the chroma — push colors away from the grey (luma) axis in an opponent / HSV space; a uniform scale over-saturates already-vivid colors and skin

• **vibrance**: a *smarter* saturation — boost **less-saturated** colors more and **protect** already-saturated ones and **skin tones** (a non-linear, hue-aware gain)

• **selective / HSL color** (Lightroom's mixer): per-hue-band hue/saturation/luminance, and **color grading** (shadows / mids / highlights tinting) from film & video

• the *mechanics* (curves, LUTs, pipeline placement) live in **Basic Image Processing** (point operations); here we own the **color-space** reasoning — which axis, which space, why protect skin

• the key framing: "convert to black & white" is **under-specified** — you are projecting **three numbers to one**, an inherently **lossy, many-to-one** map, so there are *many* valid answers and the right one depends on **intent**, not correctness

• **pick a single channel** (R, G, or B): quick; occasionally useful (the **green** channel is close to luminance and usually least noisy; the **red** channel smooths skin and darkens skies) — but it throws away two-thirds of the data

• **average** $(R+G+B)/3$: simplest, but **perceptually wrong** — it makes blue as bright as green; skies and foliage come out muddy

• **weighted luminance** (the perceptual default): a weighted sum that matches the eye's sensitivity — **green dominates**, blue counts little — **Rec. 601** $0.299/0.587/0.114$ or **Rec. 709** $0.2126/0.7152/0.0722$; this is exactly the cone-weighting / luminance idea from [[Human (and animal) vision and color]]

• **but in which space?** the recurring **[[#Linear vs gamma vs. log encoding|luma vs luminance]]** distinction — the cheap version weights the **gamma-encoded** values (**luma** Y′, what most software does), the radiometrically correct version weights **linear** RGB (**luminance** Y) then re-encodes; they give visibly different greys (linear-light is darker in saturated reds/blues)

• **lightness, not luminance**: take **L\*** from **CIELAB** (perceptually uniform lightness) — or the UI-space shortcuts **HSL "lightness"** $(\max+\min)/2$ and **HSV "value"** $\max(R,G,B)$ — each a different grey again

• **custom channel mixer** (the artistic control): choose the weights yourself (keep $\sum w_i\approx 1$ to hold exposure) — this is Lightroom/Photoshop's **B&W mixer**, and the digital descendant of the darkroom **color filter on B&W film**: a **red** filter darkens a blue sky for dramatic clouds, a **yellow/green** filter lightens foliage, an **orange** filter smooths skin (Ansel Adams's black skies are a red filter on panchromatic film). So a B&W conversion is really a decision about *how each color should map to grey*

• **the hard case — isoluminant colors collapse**: two distinct hues with the *same* luminance (a red and a green of equal brightness) map to the **same grey**, erasing the edge between them; **contrast-preserving decolorization** methods (Color2Gray, Gooch et al. 2005; Grundland & Dodgson 2007) optimise the mapping to keep **color contrast** rather than only luminance — the principled answer to "the green apple and red apple shouldn't vanish into one grey"

• dimensions of color perception:hue, chroma, etc.

• perceptual color space

• HSV and other UI-spaces

• color appearance models

• a *memory color*: viewers are unusually sensitive to skin looking right

• the skin-tone locus; cameras and film are tuned to render skin pleasingly

• diversity of skin tones and historical bias (e.g. Shirley cards) — a fairness tie-in (see ethics)

• effect on light (linear, quadratic). Why f/# are a ratio: same amount of light (divide fov, compensate with aperture)

• side effects (motion blur, depth of field). Will derive later

• **f-stops are a log₂ (doubling) scale of light**: the *f-number* itself is a **ratio** N = f/D, but what a photographer counts is **stops**, and **one stop = a factor of 2 in light**. Because light ∝ aperture **area** ∝ D² ∝ 1/N², the standard series **1.4, 2, 2.8, 4, 5.6, 8, 11, 16** steps by **√2 in N** so the area (and the light) **halves** each step. A "stop" is therefore a **log₂ unit**: $\text{stops} = \log_2(\text{light ratio})$ — and aperture, shutter time, and ISO are all measured in this same stop currency, which is exactly why you can trade one against another (the exposure triangle). Cameras add **⅓-stop** clicks for fine control. [f-stop power-of-two progression figure]

• 💡 **Big lesson:** **photographers measure light in *stops* — a log₂ (doubling) scale.** Aperture, shutter, ISO, and dynamic range are all counted in stops (factors of 2), so **exposure is additive in log space** — the same "ratios matter, work in log" principle as gamma and tone mapping.

• 🖱️ **Interactive (web edition):** a live **exposure-triangle simulator** — two 3D subjects at different depths walk in opposite directions while you set shutter, aperture, ISO, scene brightness, and sensor size; it renders the scene *physically* (time supersampling → real motion blur, aperture supersampling → real depth of field, a shot+read-noise model → real ISO grain), and an "auto-set triangle" holds the exposure constant so you feel each control's trade-off, not just its brightness. [fig-exposure-triangle-sim; interactive]

• **autoexposure & metering**: the meter integrates scene light (TTL — center-weighted / **spot** / **evaluative-matrix**) and renders it to a **mid-gray (~18%) assumption**, which **fails on high-/low-key scenes** — a snowfield meters to gray and comes out *underexposed* (you add +EV); a black cat meters to gray and comes out *overexposed*. Matrix metering (e.g. Nikon's 3D Color Matrix, trained on tens of thousands of photos) adds brightness/color/contrast/distance cues to guess intent. [18%-gray metering figure]

• **where does 18% come from?** it's the reflectance of the standard photographic **grey card** and the meter's stand-in for the *average reflectance of a "normal" scene*. The key idea: **18% is middle grey *perceptually*, not numerically** — because lightness is roughly **logarithmic** (Weber–Fechner; cf. CIE $L^*$), the perceptual midpoint between a black (~2–3% reflectance) and a white (~90%) sits near their **geometric mean** (√(0.03·0.9) ≈ 0.16), i.e. ~**18%**, *not* 50%. It is **Zone V** of Ansel Adams's Zone System (the middle of eleven zones). A meter's job is then "assume the scene averages to middle grey, and expose so that average lands at middle grey."

• 💡 **the radiometric meaning of (shutter, aperture, ISO) — a worked example (and why not to trust it).** The three controls aren't arbitrary dials: they tie the photo to **absolute scene radiometry** through two calibrated relations.

• **ISO is calibrated to the sensor** (ISO 12232). The **saturation-based speed** is $S_{\text{sat}} = 78 / H_{\text{sat}}$, where $H_{\text{sat}}$ (in **lux·s**) is the focal-plane *luminous exposure* that just **saturates** the sensor; so a higher ISO means less light is needed to fill the well. (The standard also defines SOS / REI variants tied to a mid-grey output level — which is partly why bodies disagree.)

• **the exposure (metering) equation** ties the controls to scene **luminance** $L$ (cd/m²): a "correct" (middle-grey) exposure satisfies $\boxed{\,N^2/t = L\,S/K\,}$, with $N$ the f-number, $t$ the shutter (s), $S$ the ISO, and $K \approx 12.5$ the meter calibration constant (ISO 2720).

• **worked example** — **f/2.8, 1/100 s, ISO 400.** The scene luminance this renders to middle grey: $L = K\,N^2/(t\,S) = 12.5 \times 2.8^2 / (0.01 \times 400) = 12.5\times 7.84/4 \approx \mathbf{24.5\ \text{cd/m}^2}$. The **focal-plane exposure** actually delivered to the sensor follows the *camera equation* $H = q\,L\,t/N^2$ with the lens factor $q=\tfrac{\pi}{4}T\cos^4\theta \approx 0.65$ on-axis (DxO uses $q=0.71$): $H \approx 0.65\times 24.5\times 0.01/7.84 \approx \mathbf{0.020\ \text{lux·s}}$. Sanity check against saturation: at ISO 400, $H_{\text{sat}}=78/400 = 0.195$ lux·s, so middle grey sits $\log_2(0.195/0.020)\approx\mathbf{3.3\ \text{stops}}$ below clipping — that's your highlight headroom. So a bare $(t,N,S)$ triple **does** pin down scene radiance in real units.

• ⚠️ **but two big caveats** before you believe the number. **(1) the nominal values lie** — the marked/EXIF $N$, $t$, $S$ are not the true ones (rounded shutter, manufacturer-tuned ISO that can be **⅓–1 stop** off the measured value; → [[Basic image processing and ISP#Beyond the pixels: basic metadata and EXIF]], [@burggraaff-etal-2019]). **(2) not all pixels see the same light** — lens **vignetting** and the **cos⁴θ** natural falloff (→ [[#Sensors: photosites, CCD vs CMOS]]) mean **edge pixels get less light than the centre** for the same scene radiance, so "absolute radiance per pixel" needs a per-lens **flat-field** correction, not one global scalar. Honest radiometry therefore **calibrates the actual body + lens + aperture** rather than trusting the triplet. (DxOMark's "measured ISO" is exactly this saturation-based calibration; their old *ISO-sensitivity* measurements page is now folded into the [DxOMark glossary → ISO speed](https://www.dxomark.com/glossary/iso-speed/) and the [sensor-testing protocol](https://www.dxomark.com/dxomark-camera-sensor-testing-protocol-and-scores/).) [refs: ISO 12232; ISO 2720; [@burggraaff-etal-2019]]

• caveat (the **18 vs 12.5%** confusion): light meters are actually calibrated to a fixed **luminance constant** ($K$, ISO 2720) that, with typical lens flare, corresponds to **~12–13%** reflectance — so the famous "18% card" and the meter's true midpoint differ by **~⅓–½ stop**; metering off an 18% card gives a slightly different exposure than the meter's internal target, a perennial source of forum arguments.

• **priority modes & their failure cases**: aperture-priority (set N, camera picks t) and shutter-priority (set t, camera picks N) — each fails when no valid partner exists (f/1.4 in bright sun needs an impossible shutter; 1/1000 in dim light needs an impossible aperture)

• **"expose to the right" (ETTR)**: push the exposure as bright as possible **without clipping** highlights, since the **shadows** are where noise lives; **raising ISO** beats brightening in software because ISO gain amplifies the signal *and* the pre-amplifier noise but **not** the downstream **read noise** (→ Noise) [ETTR histogram]

• the **PASM** modes are just *who picks which leg of the exposure triangle*: **P** (program — camera picks aperture+shutter, you can shift the pair), **A/Av** (you set aperture → DoF, camera picks shutter), **S/Tv** (you set shutter → motion, camera picks aperture), **M** (you set both); plus full **Auto** / scene modes

• **auto-ISO** is the modern fourth knob: in A or S the camera still has only one free leg — auto-ISO frees a *second* one by letting ISO float, so you can hold *both* a chosen aperture **and** a minimum shutter speed; you cap the **max ISO** (noise budget) and a **min shutter** (often a "1/focal-length" rule, or a factor of it)

• **exposure compensation** (±EV) biases the meter's mid-grey target — the manual override for the 18%-grey failure (snow → +EV, dark scene → −EV) without leaving an auto mode

• **M + auto-ISO** is a popular combo: you nail aperture and shutter for look, and let ISO be the automatic variable, with ±EV still steering it

• **the two classic passive AF families**:

• **contrast-detection (CDAF)**: drive the lens until image **contrast/sharpness is maximal** — accurate but has to **hunt** (it doesn't know which way is "more in focus", only that it overshot); historically slow, common on early mirrorless / compacts

• **phase-detection (PDAF)**: look at the scene through **two opposite edges of the lens** (sub-apertures) → two slightly shifted images; the **phase difference gives the direction *and* amount** of defocus in **one shot**, so the lens moves straight to focus (open-loop, no hunt). Classic SLR mechanism (a dedicated AF sensor under the mirror); it is literally a **tiny stereo measurement**

• **on-sensor PDAF / dual-pixel**: mirrorless bodies put phase detection *on the imaging sensor* — masked AF pixels, or **dual-pixel** designs that split each photosite into two halves that see opposite sub-apertures → phase detection at (nearly) every pixel, the best of both worlds (used for Pixel's portrait-mode depth too — a literal micro-stereo pair)

• **depth-from-defocus AF (Panasonic DFD)**: a third route — model how the lens's blur (PSF) changes with focus and read the **defocus direction and amount** from **two frames at slightly different focus**, giving a PDAF-like one-step cue from a plain **contrast** sensor (no phase pixels); rides on CDAF, needs a **per-lens blur profile** (full treatment → Optics, *Focus*)

• **active AF** (for completeness / history): time-of-flight ultrasonic (old Polaroid sonar) and IR rangefinding — work in the dark but limited; distinct from an **AF-assist lamp** that merely adds contrast for passive AF

• **focus modes**: **AF-S / one-shot** (lock once, for still subjects) vs **AF-C / AI-Servo** (track continuously, predict motion, for action); plus **MF** with focus aids (peaking, magnify — see UI)

• **focus selection & automation** — where computation now lives: single-point / zone / wide-area selection, then **subject-detection AF** — **face → eye → animal/bird/vehicle** detect that picks and **tracks** the subject (deep-learning driven; →ML). **Eye-AF** is the headline modern feature for portraits.

• **handling tricks**: **back-button focus** (decouple AF from the shutter button — focus with a thumb button, shutter only releases) and **focus-and-recompose** (lock focus on the subject, then reframe — beware the small focus-plane error at wide apertures / close range)

• **primes vs zooms**: a **prime** is one focal length — typically **brighter (faster), sharper, smaller, cheaper** for its quality, and "makes you think / move your feet"; a **zoom** trades some of that for **convenience** (one lens, many framings). Fredo's standing advice: a cheap **35–85 mm f/1.8 prime** is razor-sharp value; modern zooms are good but the basic kit zoom is the weak point.

• **the focal-length families** (full-frame reference; *what each is for*):

• **ultrawide** (≤24 mm): landscapes, interiors, exaggerated near/far — watch the perspective stretch

• **wide** (~24–35 mm): environmental / reportage, "the scene + context"

• **normal** (~50 mm): close to the eye's perspective, unobtrusive, fast & cheap

• **short telephoto** (~85–135 mm): **portraits** — flattering compression, easy subject separation (see the portrait-distance bullet below)

• **telephoto / super-tele** (~200 mm+): sports, wildlife, the moon — strong compression, brings far subjects close

• 🖱️ **Interactive (web edition):** a live **dolly-zoom simulator** — the camera dollies in/out while the focal length zooms to hold the subject the same size, so *only* perspective changes and the background rushes in or falls away (the "Vertigo" effect); log focal-length (12–200 mm) and distance sliders with a magnification-preserving link, realistic environment maps, and a subject that is a bust or one of several diverse 3D humans. It makes "focal length vs distance / background compression" tangible (the Pinhole chapter's lesson). [fig-dolly-zoom-sim; interactive] *(human 3D models generated with Meshy AI — acknowledge in the caption)*

• **why portrait lenses flatter — it's the *distance*, not the lens**: a "portrait" focal length (~**85–135 mm**) flatters because it lets you **back away** from the subject, and what actually shapes a face is the **camera-to-subject distance**. Up close (a wide lens), the nose is much nearer the lens than the ears, so it looms — bulbous nose, big forehead, receding ears; step back and those distances equalize, giving natural proportions. Striking finding: the chosen **distance changes not just proportions but perceived *personality*** — closer faces read as less trustworthy / less attractive in rating studies. Cite **Perona 2007** ("A new perspective on portraiture," *Journal of Vision* 7(9):992) and the perceived-traits-vs-distance study (Bryan et al., PMC4114730: <https://pmc.ncbi.nlm.nih.gov/articles/PMC4114730/>). The flip side — the **wide-angle face stretch** up close — and its correction are in [[Perspective distortion and its correction]].

• 🖱️ **Interactive (web edition):** a live **face-distortion simulator** — set the focal length, the magnification (face size), and the eccentricity (how far off-centre the face sits), and watch the face's *shape* change: the close-wide "big-nose" perspective when a short lens forces you in, and the wide-angle stretch as the face nears the frame edge. A full-frame view and a face-cropped view (constant size) let you compare shape, not position; the subject can be a face, a grid sphere, or one of several diverse 3D human faces. [fig-face-distortion-sim; interactive] *(human 3D models generated with Meshy AI — acknowledge in the caption)*

• **fast vs slow**: a **fast** lens has a **large maximum aperture** (small f-number, e.g. f/1.4) → more light + shallower DoF; a **slow** lens (f/5.6) is smaller/cheaper but light-starved. Zooms are often slow and/or have a **variable** max aperture across the zoom range.

• **the "35 mm-equiv" reminder** (cross-ref, don't re-derive): a focal length only means a field of view *relative to a sensor size*; smaller sensors **crop** → multiply by the **crop factor** to get the full-frame-equivalent FOV (APS-C ≈ 1.5×, M4/3 ≈ 2×, phones much more). See the Pinhole chapter's "35 mm" / crop-factor sidebar.

• **polarizer (circular polarizer, CPL)**: a rotatable filter passing one polarization. Because **reflection partially polarizes light** (Brewster's angle; Light & physics), rotating it **cuts glare and reflections** off non-metal surfaces (water, glass, foliage), **deepens blue sky** (skylight ~90° from the sun is strongly polarized) and **boosts saturation**. Costs ~1–2 stops; "**circular**" (a linear polarizer + quarter-wave plate) so the camera's beam-splitter AF/metering still work. **Not reproducible in post** — it changes *which* light is captured.

• **neutral-density (ND) filter**: a **uniform grey** that cuts light by a chosen number of **stops** with (ideally) no color shift → lets you use a **slow shutter** (silky water, motion blur) or a **wide aperture** in bright light. Strong NDs (6–10 stops) enable **daytime long exposures**; a **variable ND** is two stacked polarizers rotated to dial the strength.

• **graduated ND (GND)**: dark at the top, clear at the bottom, with a **hard / soft / reverse** transition. Aligned to the **horizon**, it holds back a bright sky so a high-contrast scene fits in **one exposure** — an *optical* alternative to **HDR bracketing** (Image measurements; Multiple exposure).

• **UV / clear "protective" filter**: today mostly lens protection (perennial debate: protection vs an extra surface that can add flare); historically cut UV haze on film.

• **throughline**: polarizer and GND effects are **partly or wholly un-doable in software** (the polarizer changes the captured light; the GND is a manual exposure blend) — which is why physical filters survive the computational era, in contrast to a plain ND (≈ an exposure change software *can* fake).

• a smudged or dusty front element **scatters light** → **veiling flare**, lower contrast, and soft dust spots (worst stopped down) — clean glass is real image quality, not fussiness

• **order of operations**: **blower first** (lift grit without dragging it), then a soft **brush**, then a **microfibre cloth / lens pen**, and only then **lens fluid** for stubborn smears — wipe **gently, centre-outward**; never a dry rough wipe (it grinds grit into the **coating**)

• **don't overdo it**: every wipe risks the anti-reflection coating, and a little dust is optically harmless; a **protective/UV filter** or the **lens hood** prevents most of the problem

• **sensor dust** is the cousin problem — dust on the **sensor (or its cover glass)** shows as **soft dark spots fixed in the frame** (vs moving with the lens), most visible at **small apertures** on plain areas (a clear **sky**); the cure is cleaning (rocket **blower**, then a **wet swab**) plus in-camera **dust-removal / ultrasonic shake** and **dust-mapping** for software subtraction — a maintenance reality of **interchangeable-lens** cameras (changing lenses lets dust in) (forward-ref dust-spot removal, BASIC)

• **what it does**: a gyroscope senses the camera's **angular shake** and a feedback loop counters it during the exposure, so you can hand-hold **slower shutter speeds** — typically a **gain of ~2–5 stops** (e.g. 1/100 s at 1000 mm on a monopod, per the slides)

• **three implementations**:

• **OIS — optical, in-lens**: a floating lens element shifts to counter shake (Canon IS, Nikon VR) — tuned per lens, also stabilizes the viewfinder/AF

• **IBIS — in-body**: the **sensor** itself rides a moving stage (Panasonic, Olympus, Sony) → works with *any* mounted lens, and can stack with OIS; also enables sensor-shift super-resolution / high-res modes

• **digital / EIS**: crop a margin and **warp each frame** to cancel motion — cheap, no moving parts (phones, video), but costs resolution/FOV and can add wobble

• **the crucial limit** — it fixes **camera shake, not subject motion**: a stabilized 1/4 s shot of a *static* scene is sharp, but a *moving* subject still blurs (you need a fast shutter for that). State this explicitly — a common beginner confusion.

• **sidebar — IS hacking**: the stabilizer is a controllable blur actuator — driving it deliberately enables **custom motion blur** / coded exposure (Bando–Holtzman ICCP 2011) → Advanced

• **viewfinders**: **OVF** (optical — an SLR mirror/prism or a rangefinder window) shows the real world directly, zero lag, but **not** the captured exposure/WB; **EVF** (electronic, mirrorless) and the **rear LCD** show the *sensor's* live feed — slightly laggy, but **WYSIWYG**: exposure, white balance, and depth-of-field preview all visible before the shot

• **exposure preview & the live histogram**: the EVF/LCD can show the image *as it will be captured* (exposure simulation), with a **live histogram** to judge clipping; this is the practical home of **ETTR** — push the histogram right without clipping

• **focus aids**: **focus peaking** (highlight high-contrast in-focus edges) and **magnify** for manual focus; **zebras** (animated stripes over near-clipped highlights) for exposure — borrowed from video

• the through-line: an EVF turns the *abstractions* of exposure and focus into **direct visual feedback**, which is why mirrorless changed how people shoot

• **frame rate & shutter angle**: video sets exposure time as a fraction of the frame interval — the **180° shutter-angle** convention (shutter ≈ ½ the frame time) gives "natural" motion blur; high frame rate → slow motion

• **rolling shutter / "jello"**: CMOS reads the sensor **row by row**, so fast motion (or a fast pan, or a spinning prop) **skews/wobbles** — a **global shutter** avoids it but is rarer/costlier (ties back to CMOS readout in Image measurements as integrals)

• **log / flat profiles + grading**: shoot a **log** (flat, low-contrast) profile to preserve dynamic range, then **color-grade** in post — the video cousin of raw + tone curve

• **codecs & bitrate**: intra- vs inter-frame compression, 8- vs 10-bit, chroma subsampling (4:2:0 vs 4:2:2), bitrate — trade file size vs editing/grading headroom

• *(kept deliberately brief — the algorithms and the full motion treatment are in **Motion/Video**)*

• most cameras run **closed firmware**: you cannot run your own code on them, so the **computational-photography ideas in this book can't be tried on a stock camera** — a real obstacle for experimentation

• the **hacks**: **CHDK** (Canon PowerShot) and **Magic Lantern** (Canon DSLRs) are community firmware add-ons that expose scripting, raw video, bracketing, intervalometers, and focus stacking — born exactly because the manufacturers wouldn't

• the **open road**: **Android's Camera2 / CameraX** API (and Google's earlier FCam research camera) expose per-frame control of exposure / focus / gain and access to **raw burst** — which is *why phones, not cameras, became the home of computational photography*

• the **maker road**: a **Raspberry Pi + camera module** (the HQ camera, driven by **libcamera / picamera2**) is a cheap, fully **open and programmable** camera — per-frame exposure/gain control and **raw** access on a hobbyist board — the natural platform for *trying this book's algorithms* and building DIY computational cameras

• **why this matters for the book**: our algorithms (HDR+, burst denoising, computational bokeh) need **programmable capture** — control of the per-frame integral and access to raw; the platform that grants it is where the research happens

• the **mirrorless block diagram** (light → bits): **lens + mount** (electronic contacts carry AF/aperture/IS) → **sensor on an IBIS stage** → **shutter** (a mechanical curtain *and/or* an electronic/rolling shutter) → **image processor + AF** (on-sensor PDAF, subject detect, the ISP) → **EVF + rear LCD** (the live feed) → **card + battery**

• the defining change from the **SLR**: the **mirror box and optical prism are gone** — the sensor feeds the EVF directly, which is what enables WYSIWYG preview, on-sensor PDAF across the frame, silent electronic shutter, and tighter lens-to-sensor distances (shorter flange distance → new mounts)

• each block is a previous section made physical — a good place to consolidate the chapter

• the hardware constraints: a **tiny sensor** behind a **bright, fixed-aperture lens** (no iris — aperture is fixed, exposure is shutter + ISO only); tiny sensor → **huge depth of field** (why phones must *fake* background blur) and **more noise** (why they burst-and-average)

• a **multi-camera array** stands in for a zoom: separate **ultrawide / wide / tele** modules, the long end often a **periscope** (a prism folds the light path sideways so a long focal length fits a thin body); switching "zoom" really switches cameras (+ crop/fusion between them)

• 💡 the key idea: the **computational pipeline does the heavy lifting** — demosaicking, multi-frame **HDR+ / burst denoising**, **computational bokeh** (depth from dual-pixel / multi-camera), night mode, super-resolution. The phone overcomes small-sensor physics with **computation**, not glass → forward-ref **Basic ISP** and **Multiple-exposure**.

• this is the chapter's punchline and the bridge to the rest of the book: cheap optics + a smart pipeline beats good optics alone

• **by sensor size** (the spine — sets DoF, light-gathering, and body size; cross-ref crop factor): **phone** → **1-inch compacts** → **Micro Four Thirds** → **APS-C** → **full-frame (35 mm)** → **medium format** → **large-format / view cameras**

• **by lens**: **fixed-lens** (phones, compacts, bridge/superzooms, instant) vs **interchangeable-lens (ILC)** (mirrorless, DSLR, medium format)

• **by mechanism / viewfinder**: **DSLR** (mirror + OVF, legacy) vs **mirrorless** (EVF, dominant — the anatomy above); **rangefinder** (Leica-style window + focus patch); **point-and-shoot / compact**; **bridge / superzoom** (fixed huge-range zoom on a small sensor); **action / 360** (rugged, ultrawide, fixed); **instant** (Polaroid / Instax, self-developing print); **glasses / wearable cameras** (the camera leaves your hands entirely — Google Glass, Snap Spectacles, Ray-Ban Meta, GoPro-on-a-head; **point-of-view, always-available capture**, raising the obvious **privacy / consent** questions — cross-ref Ethics); **film** bodies (still made — see the darkroom below); **view / technical cameras** (large format + movements → tilt-shift, Optics)

• **by platform — how the camera is *carried* (aerial photography)**: the **drone / UAV** is the modern default (a gimballed camera that flies), but it sits at the end of a long lineage of getting a camera *above* the scene when no tripod will reach — **balloon** photography (Nadar over Paris, 1858), **kite** aerial photography (Arthur Batut, 1880s; the rig + a timer/altitude trigger), **pigeon** cameras, and **pole / mast** photography (a camera on a tall pole or monopod for low-altitude overhead frames). The platform buys a **viewpoint**, not a different sensor — the same imaging, lifted.

• the practical lesson: **sensor size and lens system, not megapixels**, mostly separate these — the rest is ergonomics, platform, and price

• **industrial / machine vision**: defect inspection, line metrology — often **global-shutter** mono sensors with fixed lighting and GigE/USB3 interfaces; the "image" is really a **pass/fail or a measurement**

• **barcode & QR readers**: dedicated retail/warehouse scanners, and the **QR / barcode** reader built into every phone camera — the camera as a **data input** (locate → rectify → decode a 1D/2D code to a number or URL), a tiny computer-vision pipeline rather than a picture

• **robotics & drones**: cameras for **SLAM**, obstacle avoidance, **visual-inertial odometry** (camera + IMU), and grasping — output is **pose / depth / a control signal**, not a picture

• **automotive (ADAS / self-driving)**: surround cameras fused with radar and lidar for lane-keeping, sign and pedestrian detection — **high dynamic range and reliability** over prettiness

• **scientific & specialty**: microscope and telescope cameras, **high-speed** cameras, **thermal / multispectral / hyperspectral**, and **event cameras** (per-pixel brightness-change sensors) — many revisited in Advanced

• **cooled astronomy cameras**: dedicated deep-sky cameras **actively cool the sensor** (a thermoelectric **Peltier** stage, often to a fixed *set-point* tens of °C below ambient) to crush **dark current** for minutes-long exposures; typically **mono** (paired with a filter wheel — LRGB / narrowband), **regulated** so dark frames match, and **no IR-cut / no Bayer** for sensitivity — the noise chapter's thermal term, attacked with refrigeration → cross-ref *Denoising by averaging* (stacking) and the noise section

• **medical**: endoscopes, retinal and surgical cameras, capsule cameras — imaging in service of diagnosis

• **ubiquitous & embedded**: webcams and video-conferencing, **AR/VR** inside-out and eye tracking, doorbell / CCTV security, automotive cabin sensing — cheap modules, computation-heavy

• throughline (→ Intro): once the camera is a **measurement device**, "good" stops meaning *pretty* and starts meaning **accurate, fast, and robust** for the task

• the image sensor isn't the only sensor; a small suite surrounds it, and several feed the *computational* pipeline, not just the metadata

• **IMU** (accelerometer + gyroscope) — the camera's sense of its own motion; drives **stabilization** (tells OIS/IBIS which way to counter-shift), auto-rotate / level horizon, and increasingly **computation**: EIS, seeding burst/panorama alignment, gyro-based deblur (knowing the motion helps estimate the blur kernel) → Motion / Deblurring

• **context / metadata sensors**: **GPS/GNSS** (geotag → EXIF), **magnetometer/compass** (heading; AR), **barometer** (altitude)

• **ambient-light sensor (ALS)**: a small front-facing photodiode — increasingly a multi-channel **color / spectral** version — that reads the **level and color of the surrounding light** (a *non-imaging* light meter sitting beside the imaging sensor). It drives **auto screen brightness**, seeds a **white-balance prior** (what illuminant are we under? → cross-ref automatic white balance, Color technology), and a fast variant detects the **flicker** of mains-powered fluorescent/LED lighting (the 50/60 Hz pulsing) so the camera can phase its exposures to dodge **banding** (*flicker-free / anti-banding* mode)

• **dedicated focus/depth sensors**: **laser / ToF** rangefinder for fast low-light AF, **lidar** scanner for coarse depth (portrait blur, AR) — beyond on-sensor PDAF

• **microphones** (video sound; arrays → directional/wind-aware audio; sync with IMU), **housekeeping** (thermometer → dark-current/noise comp; Hall sensors → lens/stabilizer element position; real-time clock → timestamps)

• throughline: a camera is now a small **sensor-fusion** platform; the side channels (esp. the IMU) are part of *how the picture is computed*, not just notes attached to it

• **TTL metering**: the camera fires a **pre-flash**, meters the return **through the lens**, and sets flash power automatically — the flash analog of auto-exposure

• **sync speed & high-speed sync**: a **focal-plane** shutter only fully uncovers the sensor up to its **X-sync speed** (~1/200 s) — faster than that and a moving slit means the flash can't light the whole frame; **high-speed sync (HSS)** pulses the flash rapidly to cover the slit (at a big power cost). A **leaf shutter** (in-lens) opens fully at *any* speed → **syncs at all shutter speeds** (a real advantage for daylight fill).

• **fill flash**: in harsh/backlit light, add flash at **−1 to −2 EV** (the slides say −1.5 to −2, i.e. 3–4× below ambient) to **open the shadows** without looking flashed — it's **dynamic-range management**, not "more light"

• **bounce & off-camera**: never bare flash head-on (flat, harsh, red-eye) — **bounce** off a ceiling/wall (~45°) or use a diffuser to enlarge the source → softer light; **off-camera** flash gives directional, shaped light (→ Computational illumination; computational bounce flash, Davis SIGA 2016)

• 🖱️ **Interactive (web edition):** a live **portrait-lighting simulator** — three soft area lights (key / fill / kicker, each with azimuth, elevation, size/softness, intensity, and colour) on a 3D face, with **area-light-supersampled soft shadows** and a physiological melanin/hemoglobin skin model; a from-behind "setup" view shows the lights as studio umbrellas plus the camera, and lighting presets (butterfly, loop, Rembrandt, split, clamshell, rim) let you feel how light *placement and size* sculpt a face. [fig-portrait-lighting-sim; interactive] *(3D studio umbrella generated with Meshy AI — acknowledge in the caption)*

• film **development**: developer (exposed silver-halide → metallic silver) → **stop bath** → **fixer** (dissolves unexposed grains) → a **negative** (dark where the scene was bright)

• **printing**: an **enlarger** projects the negative onto light-sensitive **paper**, developed in turn → a positive print; the enlarger is a *second, fully controlled exposure*

• print controls: **exposure time** = overall lightness; **contrast grade** (paper grade / filter) = tone-mapping steepness; **dodging** = hold light back → lighter region; **burning** = add light → darker region

• the origin of the **dodge/burn** tools, the **exposure/contrast** sliders, and the **Zone System** (the 18% gray of metering = Zone V); makes the "print = performance of the negative" metaphor concrete

• **parametric / non-destructive (Lightroom style)**: the original file is untouched; edits are a re-rendered *recipe*. Global: white balance, exposure, contrast, highlights/shadows/whites/blacks, tone curve, HSL, clarity/texture/dehaze, sharpening, noise reduction, lens corrections, crop. Local: masks, graduated/radial filters, AI subject/sky masks. Plus **asset management** (catalog, rating, keyword, search) across whole shoots

• **pixel editor (Photoshop style)**: edits pixels directly — layers, selections, masks; clone stamp & healing brush, content-aware fill, dodge/burn, curves/levels adjustment layers, filters, text, montage → retouching & compositing

• the split: **global · reversible · many photos** vs **local · pixel-level · one photo**; serious workflows use both (develop in LR → retouch/composite in PS)

• 💡 the bridge: **the slider is the interface; the rest of the book is the machinery** — name each tool and forward-ref the chapter that explains it

array, basic representation and manipulation

• an image is an H×W×C array: a 2D grid of pixels, each a vector of C channels (usually R, G, B)

• sometimes extra channels, for example hyperspectral or for transparency — more in compositing chapter

• warn that it can be a problem when reading PNG if you assume just RGB and you get more channels

We'll focus on floats in [0,1]

8 bits are used on disk and for fast computation but dealing with 8 bit arithmetic is a pain. We discuss it in later chapter (ref).

• side bar: **how floating point works** — a float stores a number as **sign × mantissa × 2^exponent**, binary scientific notation: the **exponent** slides the binary point so the same ~24 mantissa bits (32-bit `float`) give **relative** precision across a huge range. What this means for images:

• **precision is relative** — fine resolution near 0, coarser for large values; a good match for **linear radiance / HDR** (and roughly for how we perceive light)

• values **outside [0,1] are fine** (super-whites, the negative lobes of a sharpening filter) — nothing clamps until you output

• `0.1` is **not exact** in binary, and adding numbers of very different magnitude **loses the low bits** (a long sum of many small values drifts) — usually negligible for images, occasionally not

• **inf** and **NaN** (`0/0`, `log` of a negative, `0×inf`) **propagate** and silently poison an image — a classic bug source; check for them

• `float32` is the working default; **`float16`/half** halves memory (ML, some HDR pipelines), `float64` is rarely needed for images

• [forward ref: the pains of **8-bit integer** arithmetic — overflow, rounding, no headroom — later chapter]

• **reminder** — 💡 **L6, quantization is rarely the real problem** (the lesson is introduced in FUNDAMENTALS → [[Image Formation and linear perspective#Noise|Noise]]): with enough bits and a sane encoding it's **noise and dynamic range** that bite, not the number of levels — which is exactly why **floats with headroom** are the sane default here.

• **how many bits for what** (match the depth to the *job*):

• **8-bit integer** — **display and delivery**, and compact storage. ~256 levels is enough *perceptually* **only because gamma** puts the codes where the eye is sensitive (the darks — see *encoding* below); but it has **no headroom** and **rounds on every operation**, so it's a poor *working* format.

• **10–14-bit** — what **camera raw** delivers, sized to the sensor's dynamic range (and to dual-conversion-gain / HDR sensors, FUNDAMENTALS).

• **16-bit integer** — the safe **intermediate for editing**: repeated operations don't accumulate visible **banding / rounding** the way 8-bit does (Photoshop's "16-bit mode").

• **float32 (linear light)** — the sane **working / HDR** format: relative precision, values allowed **outside [0,1]**, nothing clamps until output. **float16 / half** halves memory for ML and HDR pipelines.

• **the rule of thumb**: **more bits — and float — wherever you do *math* or hold *high dynamic range*; few bits only at the display / output end.** Spend precision where rounding and range actually hurt.

• 💡 **bit-depth and encoding decisions go together.** How many bits you need depends on the **encoding** you store them in, because a big part of **gamma's job is to give you better *uniform* quantization**: it spaces the available codes the way the eye perceives brightness — fine steps in the darks, coarse in the highlights — so **8 *gamma*-encoded bits** look smooth where **8 *linear* bits** would band badly in the shadows. (That is not gamma's *only* job — it also matches display response and historical pipelines, see *encoding* just below — but "make a few bits go further" is a big one.) The converse: you do **math in linear**, and linear light wants **more bits / float** to avoid that banding. So don't pick a bit depth and an encoding separately — **choose them as a pair**.

say clearly when things are encoded, when things are decoded.

KEY framing: a bare array of numbers is meaningless without knowing its encoding / color space

Discuss that different aspects of comp photo call for different encoding: linear (e.g. for dealing with physics, white balance, radiometry) vs. gamma vs. log

• rehash gamma and other encoding, ref to early section/chapter

• JPEG/film are usually some weird remapped versions for beautification

• as a result, you have more radiometric approaches to image editing/processing (starting with calibrated/linear/RAW values), vs older approaches that are more hand wavy because values were less clear

• the same three letters hide real differences: **channel order** (RGB vs BGR — OpenCV), **color space / primaries** (sRGB vs Adobe RGB vs Display P3 → Color technology), **encoding** (linear vs gamma), **range** (`[0,1]` float vs `[0,255]` uint8 vs 10/12-bit), and **premultiplied vs straight alpha**. A bare array doesn't say which — read the metadata / profile, don't assume.

• an image carries more than the pixel array: **basic metadata** — pixel **width** and **height**, channel count, bit depth, the file **name / path**, creation date

• richer **EXIF** capture metadata written by the camera: **ISO, shutter speed, aperture, focal length**, lens, timestamp, **GPS**, **orientation** flag, embedded ICC profile

• ⚠️ **EXIF is frustrating in practice**: not every camera writes every field and the set / format is **inconsistent** across makers; some values (notably **ISO** and **shutter speed**) are **approximate / rounded** and **not radiometrically correct** — don't treat them as calibrated measurements

• **concrete numbers** (so the warning isn't hand-waving): the recorded **shutter speed is the *nominal* one** — a phone that actually exposed for 1/3.0 s or 1/3.9 s may write **"1/3 s" in both cases**, a photometric error of up to **~30 %** (the kind of EXIF exposure-time rounding documented in consumer-camera calibration work, [@burggraaff-etal-2019]); and the **EXIF ISO is the manufacturer's *setting*, not the measured sensitivity** — independent RAW measurements (DxOMark, ISO 12232 saturation method) routinely find the **real ISO differs from the marked one by up to ~⅓–1 stop** (a body set to "ISO 100" may measure ~80, "ISO 3200" may measure ~2000), because makers tune the gain/headroom and the standard leaves slack. So for any **radiometric** use (HDR weights, noise modelling, exposure fusion) treat EXIF shutter/ISO as a *hint*, and prefer a per-body **calibration** or the **DNG `NoiseProfile`** when available. (Refs: [@burggraaff-etal-2019]; DxOMark *ISO speed* glossary + sensor protocol; ISO 12232.)

• a catalog of the typical fields lives in the appendix → [[Common EXIF fields]]; the file-format side of metadata is in [[File formats and compression#Data versus metadata: EXIF|File formats]]

• 💡 **what metadata *should* travel with an image** (a recurring wish in this book): beyond the basics, a sane representation would carry exactly the things a bare array can't imply — the **encoding** (linear / gamma / log), the **color space / primaries** (ideally an embedded **ICC** profile), the **noise level** (the read + shot noise parameters, so downstream code knows the per-pixel uncertainty), and — when known — the **point-spread function (PSF)** (for deconvolution / deblurring). Most formats carry the first two unreliably and the last two not at all, yet later algorithms (white balance, **denoising**, **deblurring**, HDR) silently *need* this — so an honest image is pixels **plus** this provenance.

• 💡 **noise level deserves to be metadata — and almost never is.** Of that wish-list, the **noise level** is the most useful and the most neglected: a tiny per-sensor calibration (read noise + a shot-noise/gain term, i.e. the variance as a function of signal at the current ISO) tells downstream code **how much to trust each pixel**. It is **rarely written into any image file**, yet **denoising, HDR merging, deblurring, and any weighted least-squares fit** all silently need it — without it they fall back to guessing the noise from the image. It costs only a couple of numbers to store, so carrying it alongside the pixels would make a lot of estimation principled instead of ad hoc. (**DNG**'s `NoiseProfile` tag is a rare format that actually does this.) → [[Noise, signal-to-noise ratio and dynamic range]], [[Denoising]].

• preview the taxonomy of image operations by what they touch (point / domain / neighborhood) — developed in Point operations

packing multidimensional arrays in memory. Various options.

Which one we choose (different in Python and our C++ implementation)

• channel order: RGB vs BGR (e.g. OpenCV)

• interleaved (HWC) vs planar (CHW); CHW is what PyTorch wants later

• **the C++ image data structure** (C++ profile — show this as the book's working representation): a minimal `Image` holding `width, height, channels` and a flat `std::vector<float> data` in **planar** order, indexed `data[c*width*height + y*width + x]`, with a `float& operator()(int x, int y, int c)` accessor hiding the index arithmetic. (Python profile: a NumPy `H×W×C` array.)

origin top-left (y down) vs bottom-left (y up)

pixel center vs pixel corner: the half-pixel offset — a classic bug source; matters for resampling / warping (where-from vs where-to)

• **the problem**: a `float[H][W][3]` is **mute** — it does not say whether it is **linear or gamma**, **sRGB or P3**, `[0,1]` or `[0,255]`, **straight or premultiplied** alpha. Nearly every "why are my colors wrong?" bug is an operation applied to data in the **wrong type** (e.g. white-balancing or blending gamma-encoded values as if they were linear). The cure is to make the image **typed**.

• **a good image type encodes its gamma (and the rest of its provenance)**: alongside the array, a small **type record / tags** — **encoding** (linear / gamma / log), **color space / primaries** (ideally an embedded **ICC** profile), **range and bit-depth**, **alpha kind** (straight vs premultiplied). The encoding is **part of the value, not a fact you keep in your head**. (This is the "what metadata *should* travel with an image" wish from *Beyond the pixels*, promoted into the type.)

• **conversion = cast = convert the numbers AND re-tag** (the key discipline): a `to_linear()` must **divide out the gamma *and* set the encoding field to `linear`**; a `to_uint8()` must **scale and round *and* set the range / bit-depth tag**; a color-space convert applies the 3×3 *and* updates the **primaries** tag; (un)premultiply flips the **alpha** tag. A cast that changes the numbers **without** updating the tag (or the reverse) is exactly the bug. Done right, **type and data are always consistent**, and an operation can **refuse, or silently auto-convert,** a mismatched input.

• **why it's worth the ceremony**: it makes the **illegal states unrepresentable**. White balance and exposure can *insist* on **linear** input (multiply-in-linear, [[Color technology]]); a blend can *insist* on matching **alpha kind** and **color space**; a save can *insist* on an output **encoding**. A type check catches the classic mistakes up front, instead of the *viewer* catching them later. It is **units for images**.

• **in practice**: most arrays are untyped (NumPy, OpenCV) — which is *why* these bugs are everywhere; but **color-managed** apps, Adobe's pipeline, and **DNG-style** containers ([[Appendices#DNG: the Digital Negative]]) effectively carry this type, and the book's own `imageops` treats encoding/space explicitly. The takeaway: **track the type, and let conversions move it — don't keep it in your head.**

• **RGBA**: a 4th **alpha** channel = per-pixel opacity / coverage; **premultiplied** (RGB already scaled by α) vs **straight** alpha is a classic bug — full treatment in **Compositing**

• other extra channels: a depth / Z channel, a matte, or **hyperspectral** (many narrow bands) — the array just gains channels

• **video** = an image sequence: just an extra **time** axis (H×W×C×T); most per-frame operations generalize directly (and motion couples frames — Motion/Video)

• **higher-dimensional** imaging: volumetric **3D** (medical CT/MRI: H×W×D), **4D light fields** (two spatial + two angular), spectral cubes — the same array machinery, more axes

• wrap pixel access in a **safe accessor** so out-of-bounds reads don't crash or read garbage — and decide what a pixel *past the boundary* means:

• **black / zero** (pad with 0 — darkens edges), **clamp / replicate** (repeat the edge pixel), **mirror / reflect** (fold the image back), **wrap / tile** (periodic — what the DFT assumes)

• the choice matters for **convolution, resampling and warping** at borders; mirror is the usual safe default [edge-handling modes figure]

• your favorite IDE or text editor; build, run, repeat

• the key habit: **save intermediate images to disk** — the "printf" of image code

• optional: display from code; a small UI (html?)

• 💡 **Big lesson — most debugging is tedious but simple, not clever.** **90% of debugging is methodical, not inspired**, and it comes down to two things: (1) **choose smart inputs** — small, hand-verifiable cases where you already know the right answer (constant, impulse, edge — see *Test on inputs you can verify by hand*, below); and (2) **understand the execution** — walk through it **step by step** and **output / print as many intermediate values and images as you can**. The bug almost always reveals itself the moment you *look* at an intermediate, instead of reasoning about it in your head. The rest of this section is just that lesson, unpacked.

• doubt everything; assume no piece works until you've checked it

• test incrementally — never write the whole program before testing a piece

• isolate pieces; binary-search / divide and conquer to localize a bug

• change one thing at a time

• display, don't guess: dump intermediate results instead of reasoning in your head

• to confirm a step does what you think, deliberately break it and watch the output change

• small (e.g. 3×3) synthetic images: constant, impulse, edge / half-plane (half black, half white), rectangle

• feed these to intermediate stages too, not just the final output

• each algorithm also has its *own* hand-checkable test inputs — those recipes now live as a **debug sidebar in each algorithm's own chapter**: impulse → kernel in [[Neighborhood operations and convolution]]; constant / rectangle in [[Demosaicking]]; the $\sigma_r$ limits of the **bilateral filter** in [[Bilateral filtering]]; a 1–2 px shift for **image alignment** in [[Multiple exposure imaging|Multiple-exposure imaging]]

• a seg fault usually surfaces far from its cause; suspect an array/vector index

• print the index and the array size; **assert index ∈ [0, size)** in a debug build — ties to the edge accessor in Image representation

• mostly good, not always predictable (e.g. antialiasing, bad cross-fading)

• **like all AI — sometimes remarkable, sometimes it fumbles something trivial**: reproducing **Kemelmacher-Shlizerman & Seitz's *Photobios*** ([[Many images and photo collections]]), the model autonomously nailed the *hard* parts — **face registration** from landmarks and a **dynamic-programming** path through the photo collection — yet repeatedly botched a **one-line cross-fade** `frame = (1−t)·start + t·end`, needing ~5 re-prompts to get it right. Lesson: **review even the trivial parts** — competence on the hard stuff is no guarantee on the easy stuff.

• **vibe debugging / vibe explaining**: use the LLM to explain an error, propose a minimal repro, or suggest where to look — but **verify against a hand-checkable input** (above) before trusting a fix; treat generated image code as *guilty until tested*

• our guidance: keep the LLM on a short leash for **numerically / perceptually subtle** code (filtering, resampling, color, antialiasing) where "looks plausible" ≠ correct; lean on it for boilerplate, I/O, and visualization

• 🔁 **quick reminder** (already previewed back in *Image representation → Three kinds of operation*): image operations split by **what they touch** — **point** (range; same pixel's *value* → one 1-D curve), **domain** (a *different* location → resampling/warping), **neighborhood** (a *window* → convolution/bilateral). No need to re-derive it here — this chapter just **zooms into the point branch**, the simplest, and where we start. [the three operation types figure, with the **point-operation** branch **highlighted**]

• the **histogram** (the distribution of values — its own chapter, next) is the natural companion to point ops

• develop everything on a grayscale value; for color, apply the same curve per channel (saturation is the exception)

for single channel : simple 1D input output curve. the whole operation *is* that curve (the editor's curves tool); identity = the 45° diagonal

one big question: in which units (linear, gamma, log)

Say that traditional photography has always involved controlling the mapping from scene radiance to output photo tone.

side bar: the zone system. Zones are intensity ranges. encourages photographers to think/act on mapping from scene to negative and then from negative to print. Sometimes in a local way. Lots of tools: exposure, type of film, pre-exposure, chemistry, darkroom.

• 💡 **Big lesson:** **additive vs multiplicative → choice of encoding** — ask whether a tonal op is additive (→ linear/display offset) or multiplicative (→ linear light, a sum in log); light and perception are multiplicative

• exposure is multiplicative in linear space ($L_\text{out} = 2^s L_\text{in}$, one stop = a factor of 2)

• the +1-stop curve bows upward in display space; highlights clip at 1 (unrecoverable)

• do it in linear light; the naive gamma-space multiply clips at input ½

• brightness = additive offset $L_\text{out}=L_\text{in}+b$; lifts blacks, fogs the shadows

• recommend against brightness; exposure is more radiometric (keeps black at black). Brightness survives as a legacy slider / quick shadow lift

• contrast = steepen about a pivot $L_\text{out}=g(L_\text{in}-p)+p$; gain >1 steepens and clips both ends

• big question: in linear, gamma, or log (three legitimate looks; defer the full argument to tone mapping)

• sidebar: the S-curve (sigmoid) — roll off the extremes instead of clipping

• **levels**: set the **black point** and **white point** (+ a mid-grey **gamma**) — a piecewise remap that stretches the used range to [0,1]; the workhorse first edit

• read them off the **histogram**: clip the empty ends, pull the mids with the gamma slider

• a free-form **tone curve**: drag control points, smoothed by a **spline** kept **monotone** (Catmull-Rom / monotone cubic) so tones never invert [tone-curve / spline figure]

• **sidebar — splines (the math)**: fit a smooth curve through the control points (knots) $(x_i,y_i)$ as a **piecewise cubic** — one cubic per interval, the pieces joined for continuity.

• **cubic Hermite** form on $[x_i,x_{i+1}]$ with $t=\frac{x-x_i}{x_{i+1}-x_i}\in[0,1]$: $p(t)=h_{00}(t)\,y_i+h_{10}(t)\,m_i+h_{01}(t)\,y_{i+1}+h_{11}(t)\,m_{i+1}$, with basis $h_{00}=2t^3-3t^2+1,\ h_{10}=t^3-2t^2+t,\ h_{01}=-2t^3+3t^2,\ h_{11}=t^3-t^2$. The knot **tangents** $m_i$ are the free choice — picking them is what separates the schemes.

• **Catmull–Rom** (interpolating, $C^1$): tangent = centred difference $m_i=\frac{y_{i+1}-y_{i-1}}{x_{i+1}-x_{i-1}}$ — local and cheap, but can **overshoot** (so it may go non-monotone).

• **natural cubic** ($C^2$): require continuous second derivatives and minimize total curvature $\int (p'')^2$, which gives a **tridiagonal** linear system for the knot slopes — smoothest, but still can overshoot.

• **monotone cubic** (Fritsch–Carlson): start from Hermite tangents, then **clamp** them so every segment is monotone — if a secant slope $\Delta_i=\frac{y_{i+1}-y_i}{x_{i+1}-x_i}=0$ set the neighbouring tangents to $0$; otherwise keep $(\alpha,\beta)=(m_i/\Delta_i,\ m_{i+1}/\Delta_i)$ inside the circle $\alpha^2+\beta^2\le 9$. This **guarantees no inversion**, which a **tone curve** requires (brighter in $\Rightarrow$ brighter out). [tone-curve / spline figure]

• **B-splines** are the *approximating* cousin — the curve follows a control polygon and need **not** pass through the points; smoother and easy to keep monotone, at the cost of exact interpolation.

• subsumes exposure / contrast / levels; container for the principled tone-mapping curves (next chapter)

• **sidebar — how analog photography shaped the same mapping**: long before a digital curve, photographers controlled the **scene-tone → printed-tone** mapping by physical and chemical means —

• **exposure**: where you *place* a scene luminance on the film's response (the Zone System's "place this shadow on Zone III, let the highlights fall where they may")

• the film's **characteristic (H&D) curve** and its **gamma / contrast** — chosen by emulsion *and* by **development** (time / temperature / agitation / developer; **push / pull**; "**N / N+1 / N−1**" development to expand or compress the scene's range onto the film)

• **pre-exposure / flashing**: a uniform sub-threshold exposure that lifts deep shadows above the film's *toe* (raising the floor without touching highlights)

• at print time: the **paper grade** (contrast grade 0–5), the print developer, and local **dodging & burning** (hold back / add light per region)

• the free-form tone curve (and the per-region masks of *Local point operations*, below) is the **digital descendant** of this whole toolkit. *Cite **Ansel Adams**' trilogy — **The Camera**, **The Negative**, **The Print** — and the **Zone System** (developed in *The Negative*).*

• the **basic color edits** every editor exposes — the color analogue of the tone curve. **This is the home for the implementable mechanics** (the equations below); the *perceptual-space* nuance — which space to scale chroma in, why skin needs protecting — is the job of [[Color technology]] (don't defer the equation there)

• **saturation**: split each pixel into **luminance** $Y$ and **chroma** $(\text{RGB}-Y)$, scale the chroma, recombine — $\text{RGB}' = Y + s\,(\text{RGB}-Y)$ (so $s=0$→grey, $s=1$→identity, $s>1$→more saturated; equivalently $(1-s)\,Y + s\,\text{RGB}$). A *uniform* $s$ over-saturates already-vivid colors and skin

• **vibrance** (implementable): the same operation but with a **per-pixel gain** that fades as the pixel's own saturation rises, and protects skin — replace the constant $s$ by

$$g = 1 + v\,(1-\text{sat})\,w_\text{skin}(\text{hue}), \qquad \text{RGB}' = Y + g\,(\text{RGB}-Y)$$

where **sat** is the pixel's current saturation (e.g. HSV $S = \tfrac{\max-\min}{\max}$, or chroma magnitude / luma), **v** the vibrance amount, and **w_skin(hue)** a hue weight $\approx 0$ on skin / orange hues and $1$ elsewhere — so muted colors get the full boost while vivid colors and skin barely move (set $w_\text{skin}\equiv 1$ for the skin-agnostic version)

• which space to scale chroma in (perceptually uniform), and why skin needs protecting — deferred to [[Color technology]]

• **LUT (look-up table)**: bake any color/tone mapping into a precomputed table indexed by pixel value, so applying it is one **lookup per pixel** — fast, and the standard way to ship a *look*

• **1-D LUT** = three independent per-channel curves (exposure / contrast / levels / gamma collapse into one); **3-D LUT** = a full RGB→RGB map that handles **cross-channel** effects a 1-D LUT can't (white balance, film/grade looks — the `.cube` files of video), at the cost of a coarser sampled grid + interpolation

• LUTs are also a **speed** trick: precompute an expensive per-value function once, then index it (ties to fixed-point / 8-bit pipelines)

• a **separate craft** from stills / radiometric editing — the *look* of motion pictures, with its own tools and vocabulary; less radiometric, more color craft

• the colorist's primaries: **lift / gamma / gain** wheels (shadows / midtones / highlights, each with a color offset); **primary** (whole-frame) vs **secondary** (qualified by hue / luma, or masked) corrections

• the working space is usually a camera **log** encoding (S-Log, Log-C) under a color-management pipeline (**ACES**); the finished *look* is shipped as a **3-D LUT** (`.cube`), applied per frame (ties to [[#Lookup tables]])

• the **teal-and-orange** look: push skin into warm **orange**, everything else toward its **teal** complement — complementary contrast that makes people pop; popularized by ***O Brother, Where Art Thou?*** (2000 — among the first features with a full **digital intermediate**; Coen brothers, cinematographer Roger Deakins)

• the **stills-side cousin**: Lightroom's HSL / color-mixer and color-grading panels

• ties: [[#Lookup tables]] (3-D LUTs are the delivery format); [[Color technology]] (color management, log / ACES); tone mapping (the dynamic-range side)

• strictly not a point operation: apply a tool through a **selection mask** (the result depends on *where*, not just the pixel's value)

• **selections**: by value / color (magic wand), by **gradient** (linear / radial), by **brush**, **edge-aware** (snap to edges → Big lesson: edge-preserving = affinity), and increasingly **AI / semantic** (segment-anything-style)

• 💡 **the moment a point operation goes *local*, it raises the central question of "*which pixels?*" — segmentation and matting.** A hard binary **selection** is a *segmentation*; a soft-edged one (feathering, hair, motion blur, partial coverage) is a **matte** (a per-pixel α, exactly the alpha channel of [[#Alpha and extra channels]]). So even a humble masked exposure tweak is the front door to a deep topic — how to decide the mask, and how to make its border *soft and correct*. **Forward ref → [[Compositing, segmentation and matting]]** (the full treatment: matting equation, trimaps, alpha estimation), with edge-aware mask refinement in [[Guided image filtering]] / [[Bilateral filtering]].

• the bridge to **local tone mapping** and edge-preserving filtering (next chapters)

• the **histogram** = the distribution of pixel values; reading it tells you exposure, clipping, contrast, and where the tones sit [histogram figure]

• the on-screen **RGB and/or luminance histogram** in the camera and in editing apps; a **live histogram** in the viewfinder / on the rear screen while composing

• **clipping warnings** ("**blinkies**"): blown highlights and crushed shadows flash to flag lost detail at each end

• ⚠️ the displayed histogram is typically of the **gamma / display-encoded** image, **not** the linear data — so its shape (and where mid-grey sits) reflects the encoding, not scene radiance (ties to *depends on the encoding space*, below)

• beyond per-channel: a **3-D color histogram** = the distribution of pixels in **color space** (a cloud / occupancy in the color volume)

• its shape **depends on the color space** you build it in — RGB cube vs a perceptual space (Lab / opponent) bin the same pixels very differently

• used for **palette extraction, color transfer**, and **segmentation** (clusters in color space)

• its **shape depends on the encoding space** — the *same* image has a very different histogram in linear vs gamma vs log [histogram-encoding-spaces figure: linear piles into shadows, gamma spreads it, log spreads shadows more; 18% gray lands at 0.18 / ~0.46 / ~0.75]

• warning: it's a **sampled / discretized** estimate — spikes and gaps are artifacts of **binning / quantization**, not the scene

• **histogram equalization**: use the image's **cumulative histogram (CDF)** as the tone curve → flattens the histogram, maximizing global contrast (often too aggressively) [histogram-equalization figure: the CDF *is* the transfer curve]

• pseudocode: bin → running-total CDF scan → normalize → apply as a per-pixel LUT

• **histogram matching** (a.k.a. histogram **specification**) generalizes equalization: instead of flattening to a *uniform* target, reshape an image's histogram to match **any target distribution** — a chosen shape, or *another image's* histogram

• **the recipe = compose two CDFs**: equalize the source (push it through its own CDF, $\text{CDF}_\text{src}$, to a uniform), then **"un-equalize" through the target's inverse CDF** — $L_\text{out} = \text{CDF}_\text{tgt}^{-1}\big(\text{CDF}_\text{src}(L_\text{in})\big)$; both are 1-D LUTs, so matching is still a single **point operation** [histogram-matching figure: source histogram + target histogram → matched result + the composed transfer curve]

• **equalization is the special case** where the target is uniform ($\text{CDF}_\text{tgt}=$ identity) — so matching, equalization, and a hand-drawn levels curve are one family

• **for color**: matching each channel independently **shifts hues** (the channels aren't independent), so do it in a **decorrelated / perceptual** space, or use the proper multi-dimensional version — **color transfer** (Reinhard et al. 2001: match the **mean and standard deviation** per channel in the decorrelated $l\alpha\beta$ space), with the exact N-D form via **optimal transport** (Pitié et al.)

• 💡 forward ref: histogram / color matching is the workhorse behind **harmonization** (make a pasted region's tones match the destination — Compositing), **color grading / style transfer** (impose a reference look), and **texture synthesis** (match value statistics) — all instances of "make this image's statistics look like that one's"

• refs: Reinhard, Ashikhmin, Gooch & Shirley 2001 (color transfer between images); Pitié, Kokaram & Dahyot 2007 (automated N-D color transfer / optimal transport); Gonzalez & Woods (histogram specification — textbook)

• **Greg Ward's constrained / histogram-based tone mapping** (1997): equalize, but **cap the local slope** so contrast is never exaggerated beyond what the eye accepts — a perceptually-bounded version for HDR [Ward et al. 1997]

• **dynamic range** = ratio of brightest to darkest (contrast, not exposure); scene ~1:100,000, sensor ~1:1000, display ~1:20–500

• two problems: **capture** (merge multiple exposures → HDR chapter) vs **display** (squeeze a huge range onto the screen *while keeping detail* — this chapter); can't just rescale (shadows collapse)

• sidebar: limited range can be a virtue (W. Eugene Smith; lighting reduces range at capture)

• **global** = one curve everywhere (a point operation); simple, fast, no halos, but a flattened slope loses local contrast

• **local** = curve varies spatially (a neighborhood operation); split large-scale vs detail, compress only large-scale — needs an edge-aware blur [global-vs-local figure]

• naive clip / rescale both fail (linear answer to a multiplicative problem)

• the **Reinhard** global operator `L_out = L/(1+L)` (or a white-point variant) maps `[0,∞) → [0,1)` — a principled, halo-free default global tone map [Reinhard curve figure]

• **one curve for the whole image** (Reinhard's `L/(1+L)`, a log/gamma, an S-curve) — simple, no halos, but can't squeeze a huge range into a small one without flattening local contrast

• apply to **luminance** only, rescale RGB by the ratio (keep hue)

• family: gamma, log, S-curve, the freehand tone curve; Reinhard is the principled, parameter-light member

• **histogram equalization** is itself a tone map: CDF as the curve, data-driven, max global contrast — but no restraint (exaggerates busy regions, amplifies noise)

• **Ward et al. 1997**: cap the local slope → perceptually-bounded equalization for HDR

• the curve **varies spatially** (compress the slow, large-scale variations, keep local detail) — needs a **blur / edge-aware** step, so it waits for convolution & edge-preserving; its failure mode is **halos** around strong edges [local-tone-mapping halo figure]

• best-of-both decomposition: large-scale layer + detail layer; compress only large-scale, add detail back

• naive Gaussian blur crosses edges → contaminated large-scale layer → **halo** band at strong edges; fix = edge-preserving (bilateral) blur

• 💡 **Big lesson:** **edge-preserving = affinity** — weight a blur by value affinity as well as spatial distance (the bilateral filter)

• **log encoding pays off here**: range compression is multiplicative, so log turns it into a simple subtraction (→ Big lesson: additive vs multiplicative)

• detail = log(L) − large-scale (a subtraction); compress + recombine = additions

• 💡 **Big lesson:** **additive vs multiplicative → work in log** — light/exposure/range are multiplicative; log turns them into addition, which blurs/decomposes/compresses cleanly

• Ansel Adams' **Zone System** = global tone mapping: carve light into 11 zones (a stop apart), Zone V = 18% grey; place a scene tone in a zone, realize it through exposure/development/paper — the analog of Reinhard's white point [Zone System strip figure]

• side bar: **dodging & burning** — the darkroom ancestor of local tone mapping (hold back / add light locally), even local contrast / sharpness

• **video grading evolved separately** — less radiometric (lift/gamma/gain), rich craft; same operations, different dialect → see [[#Video color grading]] (Point operations)

• ⚠️ honest note: **full HDR tone mapping is not very commonly used in practice** — it's **hard to control** and tends to look **artificial / "HDR-look"** (haloes, flat, garish)

• most photographers prefer **simpler exposure / contrast adjustments** and **exposure fusion** over an aggressive HDR operator

• recap: tone mapping = display-side range compression; global (halo-free, loses local contrast) vs local (crisp detail, needs edge-preserving blur, risks halos); all cleanest in log; HDR chapter picks up the threads

• a pixel computed from its neighbors (recall the three operation types)

• blur also comes from optics: diffraction, spherical aberration → motivates removing optical artifacts (forward ref: deblurring) so important to understand if we want to prevent and remove it.

• blur as a weighted average of neighbors.

• linear shift-invariant (LSI) filtering; why linearity matters (with a non-shift-invariant counter-example)

• the convolution algorithm: loop over the output

• point spread function (PSF) / impulse response

• **debugging / hand-checkable inputs**: an **impulse** reads off the kernel (the PSF); a **constant** must pass through a normalized kernel unchanged; a **shift kernel** just translates the image; a **half-plane** (edge) shows the edge response; verify a **1-D 3×3** by hand first

• without the flip it's called correlation. USeful too but not as useful and doesn't have all the good mathematical symmetries.

• sidebar: convolution in probability (sum of random variables, Laplace first derivation of convolution, flip is obvious: if we want the sum to be a given number, then the second number has to be the sum minus the first one)

• **sidebar — debugging convolution** (recipe moved here from Developing, testing & debugging): convolve an **impulse** and the output **is your kernel**, laid out and flipped — read it off number-for-number to catch a wrong weight, an off-by-one in centering, and the **flip** (convolution vs correlation) at a glance; a **constant** comes back unchanged iff the weights sum to 1; an off-centre single-1 **shift kernel** translates the image by exactly that offset (tests sign & direction); a **half-plane** shows the border behaviour; do the very first test in **1-D (1×3)**, where every output is hand-computable. [fig-psf-impulse]

• **normalization**: the weights sum to 1 to preserve brightness (a blur shouldn't darken the image)

• **commutativity / associativity**: I*g = g*I and (I*g)*h = I*(g*h) — so filters can be pre-combined and reordered (the flip is what buys these symmetries)

• **linearity**: scales and superposes — the very reason Fourier will help (next chapter)

• the **identity** is the impulse δ (I*δ = I); **separability** gets its own point below

• edge padding (ties to the edge/accessor policies above)

• centering the kernel

• box filter

• Gaussian (formula, truncation at ~3σ)

• a 2-D filter is **separable** if it's an outer product of two 1-D filters (the Gaussian is): blur **rows, then columns**

• big **cost** win: a (2r+1)×(2r+1) kernel costs O(r²) per pixel; done separably, **O(2r)** — the practical reason large Gaussian blurs are fast [separability figure]

• **derivative filters**: finite differences estimate ∂I/∂x and ∂I/∂y; the **Sobel** kernel pairs a derivative (one axis) with a little smoothing (the other) for noise robustness

• the **gradient** ∇I = (Iₓ, I_y) → its **magnitude** (edge strength) and **orientation** (edge direction): the basis of edge detection (forward ref: features / Canny) [gradient / Sobel figure]

• unsharp mask (add back the high frequencies)

• sidebar: film photography unsharp mask.

• linear unsharp masking has two failure modes: it **amplifies noise** (in flat areas the high frequencies are mostly noise) and **haloes** at strong edges

• **adaptive / non-linear** unsharp masking **gates** the boost — sharpen **more in detailed / edge areas, less in smooth ones** (by local **edgeness** or **intensity**); cubic / rational variants (Ramponi) and adaptive control (Polesel et al.) — the principled version is in the edge-preserving part

The canonical reference is Polesel, Ramponi & Mathews, "Image Enhancement via Adaptive Unsharp Masking," IEEE Trans. Image Processing 9(3):505–510, 2000, whose whole point is that an adaptive filter controls the sharpening so enhancement happens in high-detail areas and little or none in smooth areas. Its predecessor is Ramponi, "A cubic unsharp masking technique for contrast enhancement," Signal Processing 67:211–222, 1998, which introduced the nonlinear (rational/cubic) variant. Also worth citing in this cluster: Russo, "An image enhancement technique combining sharpening and noise reduction," IEEE Trans. Instrum. Meas., 2002. [port + 3](https://baldigital.port.ac.uk/?p=5058)

say we'll have more in the edge preserving part

• a Gaussian blur **smears edges** because it weights neighbours by spatial distance alone (a near-but-different pixel pollutes the average)

• fix: weight each neighbour by **value similarity** too → the **bilateral filter** (Tomasi & Manduchi 1998): spatial Gaussian × range Gaussian on the intensity difference [edge-preserving figure]

• per-pixel normalization (weights depend on the values); **nonlinear** and **not shift-invariant** — leaves the clean world of convolution (no commutativity, no Fourier diagonalization)

• 💡 **Big lesson:** **edge-preserving = affinity** — use the color/intensity difference as how much two pixels "belong together"; the engine of edge-aware denoising/sharpening, halo-free local tone mapping, matting, colorization

• two kinds of weighted average: fixed weights → convolution (linear, shift-invariant); image-dependent weights → edge-preserving (bilateral, nonlinear)

• two threads forward: **Fourier** (next — convolution becomes per-frequency multiplication, enables deblurring) and the **affinity** lesson (recurs all book)

Why? Allows us to use linear algebra, e.g. to solve and understand inverse problems such as deblurring.

cite Strang.

Useful both conceptually (translation to the language of linear algebra) as well as practically (use linear algebra libraries and well-established solvers)

Similar to how we store in memory: serialize all pixels/channels.

Note: one source of notation overload: images are often named x.

Operations on vector spaces and what they mean for images: addition, multiplication by scalar

dot product aka scalar product aka dot product aka inner product

Matrices without forming matrices

Once images are modeled as vector, linear operators (exposure, convolution) can be represented as matrices. But giant matrices. In a case like convolution

And just like with any vector space we can do a change of basis. See next.

Convolution is a linear operation. It can be represented as a matrix. But matrices and linear operations can be unwieldy to understand. For example, we may want to know how easy it it to invert a matrix (if we want to deblur).

Said differently, the output of a convolution for a pixel depends on other pixels and these cross-dependencies create complexity.

However, there is a special case of matrix that is easy to manage : a diagonal matrix. In a diagonal matrix, there is no cross-talk, each input just undergoes a scalar multiplication, which means the inverse is a simple scalar division. The difficulty of inversion boils down to how close the scalar is from zero.

Fortunately for us, we can do a change of basis (change of image representation) so that convolution turns into a diagonal operation. That change of basis is the Fourier transform. Warning: the only bug is it assumes that the image is cyclic. But still insightful.

And we have **already seen** that frequency is a good way to model **human vision**: the contrast sensitivity function (**CSF**) back in the perception chapter (Spatial vision) describes the eye's sensitivity as a function of *spatial frequency*. So the Fourier view is not only a computational convenience for convolution — it also matches how the visual system is organized, which is why frequency keeps coming back (JPEG, image-quality metrics, tone mapping). → see Human perception → Spatial vision / CSF.

• **sidebar — a Fourier transform you can see** (motivation: it's physical, not just an imposed trick): the little **starburst** a distant point of light (a star, a streetlight) makes through a small aperture/lens is ≈ the **Fourier transform of that aperture** (squared magnitude) — round hole → **Airy** disc + rings; stopped-down iris → **sunstar** spikes (one streak ⟂ each blade-edge). The optics performs the transform **for free**, in the instant light crosses the lens — no FFT. Lesson: Fourier is a *natural, physical* way waves + apertures organize themselves; we're borrowing it for images. → ties to [[06 Optics, lenses, and aberration correction]] (diffraction / the **MTF** = FT of the PSF).

We focus on greyscale but it's the same for RGB.

Fourier transform: instead of one value per pixel, one coefficient per sine wave.

It's orthonormal, so coefficients are just dot products (sum/integrals)

• complex exponential carries a cosine + sine per frequency (Euler $e^{i\theta}=\cos\theta+i\sin\theta$); real part = cosine content, imag = sine — needed to capture any phase

• ⚠️ **so many Fourier conventions**: complex-exponential vs real sin/cos form; whether the $2\pi$ lives in the exponent, in the frequency, or in the normalization; continuous transform vs Fourier series vs DFT; normalization ($1/N$, $1/\sqrt N$, or none — and which side carries it); sign of the exponent ("forward vs backward"); angular vs ordinary frequency. These differ across textbooks, NumPy, and signal processing — **always check which convention a source uses** before you trust a formula. [fig-fourier-conventions — author's own "Fourier flavors" convention-warning illustrations, shown **full-width** so each reads clearly; from Fredo's slides 3_Linear_image_processing, slides 67–68 (author-created images)]

Figure: show change of basis matrix [⚠️ legibility: text labels currently unreadable — enlarge / redraw]

In general, Fourier diagonlizes all shift and shift-invariant operators.

• 💡 **Big lesson:** **diagonalize when you can** — express a linear operator in a basis where it's orthogonal/diagonal; **Fourier diagonalizes convolution**, turning it into per-frequency multiplication.

• **sidebar — Fourier and the heat equation** (where this trick came from): Fourier (1807) was solving **heat diffusion** in a 1-D rod with some initial temperature profile. His move was to write the temperature as a sum of **cosines** — and in that basis the heat equation **diagonalizes**: each frequency evolves on its own, and a single cosine simply **decays exponentially in time** (∝ e^{−k²t}), the highest frequencies fastest, so the profile smooths out. The cosines are the **eigenfunctions** of the Laplacian — the same reason Fourier diagonalizes convolution here.

• the heat equation also hands us the **two canonical solution families of linear differential equations: sine waves (in space) and exponentials (in time)** — separation of variables gives **spatial sinusoids** whose **amplitudes decay exponentially**, $\propto e^{-k^2 t}$, the highest frequencies dying fastest, so the profile smooths out. [fig-heat-diffusion-1d — a 1-D temperature profile (amplitude on y, position on x) smoothing over time: shown for (a) a general signal and (b) a single sine wave that simply decays in place]

• the general lesson: **diagonalizing is the win for any linear ODE / PDE** — in the eigenbasis a coupled system decouples into independent scalar equations whose solutions are **exponentials** (e^{λt}) or, for oscillatory / conservative systems, **sine waves**. Find the right basis (the eigenvectors) and a hard differential equation becomes trivial per-mode arithmetic.

• feed a pure sine into any blur → the **same sine**, only scaled and phase-shifted: sines are the **eigenvectors** of convolution, eigenvalue $\hat g(\omega)$ = the kernel's FT [sine-eigenvectors figure]

• blur leaves low freqs untouched, crushes high ones → blur is a **low-pass filter**

• the **convolution theorem** $\mathcal{F}\{I*g\}=\hat I\cdot\hat g$: convolution in space = multiplication in frequency

Give the example of a 2-pixel image visualized as a2D Euclidean plane. Our linear convolution has coefficients 0.8, 0.2 The Fourier transform is just along [1, 1] and [1, -1] (rotation by 45 degrees). Our simple convolution becomes diagonal

• 2-D: two frequency indices (u,v); display as **magnitude** + **phase**

• **symmetries**: a real image has a **conjugate-symmetric** spectrum (half the coefficients are redundant); low freqs at center, high at corners; natural images mostly low-freq; a shift → a **phase ramp**

• **magnitude vs. phase** — how to *read* an image's FT: the **magnitude** shows which frequencies / orientations are present; the **phase** carries the *structure*. Swap two images' magnitude and phase and you recognize the one whose **phase** you kept. [magnitude/phase figure]

• **compact in space ⇔ spread in frequency** — a narrow Gaussian has a wide spectrum; the locality / uncertainty tradeoff that recurs in pyramids & wavelets [compact-space-vs-frequency figure]

• sidebar: **MTF** for lenses: the optical system's frequency response is the FT of its PSF (ties to the diffraction limit, Optics)

• before we lean on it, two honest caveats that the rest of the book keeps bumping into:

• **the DFT treats the signal as cyclic / periodic**: it tiles the image and wraps around, so content leaking off the right edge re-enters on the left → **boundary leakage / ringing**. This is what motivates **windowing** before an FFT, and it is exactly the **periodic-boundary assumption** baked into the FFT deblur solver (forward ref: the FFT-based deconvolution in [[Computational tools]] inherits this wrap-around)

• **the basis has infinite / global support**: every coefficient is a **global wave** spanning the whole image, so there is **no spatial localization** — move a single pixel and *every* coefficient changes. A Fourier coefficient can tell you *which* frequencies are present but not *where*. This is precisely the motivation for **wavelets / pyramids**, which trade a little frequency precision for spatial locality → [[#Linear pyramids and wavelets]]

example of spatial aliasing. Show classical sine wave (prudential tower photo downsampling). Intuitively we need to break down the content that cannot be represented (return grey instead of black or white). You can think of it as the average over a pixel. More formally, we need Fourier.

As it turns out, aliasing too is easy in Fourier (because sampling grid is essentially a set of shifted spikes)

• **Nyquist–Shannon**: to represent a signal you must sample at **more than twice** its highest frequency; above that, high frequencies **fold (alias)** down to low ones — the moiré / wagon-wheel effect [aliasing figure] [📺 embed aliasing/sampling explainer → https://www.youtube.com/watch?v=yr3ngmRuGUc, see [[Video Resources]]]

• anti-aliasing: low-pass (pre-filter) before you downsample

• **sidebar — Aliasing on purpose: dynamic moiré prints**: aliasing / moiré is usually a nuisance, but it can be **engineered**. A printed **micro-pattern** photographed by a camera produces a moiré whose appearance **encodes information** — the relative camera pose / motion — because the beat pattern is exquisitely sensitive to the sampling geometry. Cite MIT's **KinéCam** dynamic prints (https://hcie.csail.mit.edu/research/kinecam/kinecam.html). Ties to **structured / coded patterns** elsewhere in the book.

• the space-vs-frequency tradeoff (sharp in one ⇒ spread in the other)

• sidebar: a primal-domain (no-Fourier) perspective too

• perfect sinc vs practical filters. (more in resampling section)

We will discuss more in later chapter, but quickly show:

sharp input image

blurry image (say with box. Maybe just 1D ?) plus noise.

linear inversion with amplified noise.

Fourier MTF

domain operations, function from plane to itself

start with example of scaling up (upsampling). Missing values. Say downsampling will be a bit different (too many values) but the solution will be surprisingly similar.

start naive : loop over input pixels. Point out problem (gaps)

better solution: loop over output (inverse mapping: for each output pixel, inverse-transform and sample the input)

• where from vs where to: always inverse transform

challenge

derived with intuitive 1D linear interpolation

say in2D it can't be purely linear unless we triangulate

sidebar: smart triangulation where we pick the best diagonal for each square (the one joining the two most similar values)

Say we do linear horizontally first, then vertical

exercise : show it's the same if we do it in the opposite order.

• pseudocode: upsample = loop over output, inverse-map to fractional coords, bilinear-blend the four corners (two horizontal blends then one vertical)

say that the interpolation perspective is fine for linear and upsampling, but because this is about (re)sampling, Fourier has better insights. In practice, means reconstruction through convolution.

side bar: proper resampling analysis: start with sampled signal, reconstruct, prefilter, resample. The final filter is a combination of prefilter and reconstruct. Depending on whether we up or downsample one of them dominates.

• warning: much about bicubic wrong on the internet. Including in old python libraries

They all approximate a perfect sinc but with a finite support and try to find a tradeoff between loss of medium frequency and aliasing of high frequencies. Show in Fourier : perfect box, smooth approximation

Netravali Mitchell.

form of bicubic.

required symmetries, remaining two parameters

perceptual study.

point out that photoshop gives you a choice.

The key is how you scale the filter.

• 💡 **Big lesson:** **prefilter before you downsample** — when you throw samples away, first blur / area-average out the detail too fine for the new grid, or it **aliases** into bold false low-frequency moiré; naive nearest-pixel decimation is the classic bug (→ **L16**)

• for a **shear / rotation / homography** the ideal prefilter is **spatially varying and anisotropic** — an output pixel back-projects to a stretched **ellipse** in the input, so exact filtering is costly; **mip-maps** and **EWA** (elliptical weighted average) approximate it

• side bar: **Andrew Adams' challenge** — rotate an image by **1° three-hundred-sixty times**: a naive resampler blurs it to mush (errors compound), a good one (proper prefilter + reconstruction) returns nearly the original — a stress test for resampling quality

• recap: resampling = reconstruct-then-resample, reconstruction = convolution with a finite-budget sinc approximation

• forward: the reduce step → Gaussian/Laplacian pyramids (next); inverse-mapping → warping & morphing; resampler quality → image metrics

• a multiresolution representation — **halfway between primal (space) and Fourier (frequency)**: localized in *both*, which is exactly why it's so practical

• multiresolution, coarse-to-fine (original papers, 1980s)

• repeatedly **blur then downsample by 2** → a tower of ever-coarser, smaller images (each ¼ the pixels): the **reduce** operator

• cheap (a geometric series → only ~4/3 the original pixel count in total); the basis for coarse-to-fine search and **scale space** [Gaussian pyramid figure]

• construction: each level = **G_k − expand(G_{k+1})** (the detail the coarser level missed) — a **band-pass** image [Laplacian pyramid figure]

• reconstruction: **exactly invertible** — add the levels back coarse → fine (an **encoder → decoder**) [encode/decode figure]

• pseudocode: Laplacian build (reduce + difference loop) and collapse (expand + add loop), each the exact inverse of the other

• frequency intuition: each level holds a **band** of detail, coarse → fine [frequency-bands figure]

• properties: **decorrelated** and **sparse** on natural images (mostly near-zero → compresses, denoises) — sparsity drives compression (JPEG 2000) and denoising (coring)

• **overcomplete** (>1 coefficient per pixel — a frame, not a tight basis); only **approximately shift-invariant**; **isotropic** (no orientation) — each limitation motivates a relative

• **wavelets**: a true tight basis (one coefficient per pixel, no redundancy) → natural for compression (JPEG 2000); but less Fourier-friendly, not shift/rotation-invariant

• sidebar: **Fourier, wavelets, and the locality dial** — Fourier (all frequency) ↔ primal grid (all space); wavelets/pyramids in between, a tunable space-vs-frequency tradeoff

• **steerable pyramids** (Simoncelli & Freeman 1995): add **orientation** (oriented sub-bands), nicely shift/rotation-invariant, even more overcomplete — texture, video magnification; (**multigrid** in passing)

• multiscale / **pyramid blending**: blend each Laplacian level under a **Gaussian-blurred mask**, recombine → seamless joins (the apple/orange) — see compositing & pano [pyramid blending figure]

• exposure fusion, HDR+ tone mapping — see HDR

• pyramid **denoising / coring**: zero out the small detail coefficients (mostly noise), keep the large ones — see Denoising [coring figure]

• local Laplacian filtering (edge-aware detail) — see edge-preserving

• coarse-to-fine (optical flow), scale (SIFT), pooling / U-Net (deep nets) — forward refs

• **full-reference**: compare against a known-good reference (denoising / compression / resampling) — most of this chapter

• **no-reference / blind**: score a single image with no reference (needs a model of natural images) — mostly set aside

• **MSE / L2** = $\frac1N\sum(I-J)^2$, simple, cheap, differentiable, but **doesn't match perception**: a tiny shift, a small gamma change, or a touch of noise can give the *same* MSE as a perceptually-awful edit (treats the image as a bag of numbers, ignores edges/texture/faces)

• sidebar: why squared (L2 vs L1) — smooth at 0, punishes outliers; L1 more robust

• **PSNR = 10·log₁₀(MAX²/MSE)** (dB) — a monotone re-expression of MSE; bigger is better (~30–40 dB decent); inherits MSE's blind spots

• sidebar: at least measure it in a **perceptual color space** (Lab / Luv, ΔE) so error ≈ perceived difference

• **SSIM** (structural similarity): compares **luminance, contrast and structure** in local windows — far better correlated with perceived quality; the *same* PSNR can have very different SSIM [SSIM-vs-MSE figure]

• **visible-difference predictors**: a full perceptual model (the **CSF + masking** from the perception chapter) that predicts *where* a difference is visible; **HDR-VDP** extends it to HDR

• forward ref: **LPIPS, DreamSim** — deep-feature distances → [[Deep learning]]

• forward ref: a differentiable metric is also a **training loss** — minimizing LPIPS/perceptual distance (not per-pixel MSE) is how learned methods get **sharp** results → [[Deep learning#Learned perceptual metrics and losses]]

• no single best metric: choose by what you're evaluating (fidelity vs perceived quality vs a training loss)

• sources: photon/Poisson, read, thermal (see noise / image formation)

• look at it: histogram of a flat grey patch, a scanline

• reminder: affine model (in linear space) is good.

• reminder: sensor saturates on both ends

• 1 → 3 → 5 images: the noise visibly drops [averaging-convergence figure]

• the statistics: independent noise averages out — variance ∝ 1/N, so **noise drops as 1/√N**; SNR, PSNR improve accordingly

• needs **alignment** first (brute force; → burst / HDR+, see multiple-exposure)

• no other measurements of the same pixel → use redundancy *inside* the image: neighbours are near-repeated measurements (same 1/√N win), but averaging smears edges

• everything below = a different answer to "which neighbours count as the same value?"

• spatial averaging: Gaussian blur — and why it **smears edges** [denoising before/after figure]

• bias–variance trade-off: wider filter cuts more noise but blurs more

• **median** filter: ignores the odd wild neighbour; good against speckle / salt-and-pepper

• sidebar: optimal *linear* filter = **Wiener** (per-frequency keep ∝ signal/noise) — proof we must leave fixed linear filters

• bilateral filter (edge-preserving) — see edge-preserving chapter — weight a neighbour by spatial *and* value closeness; per-pixel normalization, non-linear, not shift-invariant; color = distance in color space

• 💡 **Big lesson:** **edge-preserving = affinity** — use the **color / intensity difference** between two pixels as how much they "belong together"; the bilateral filter weights neighbours by that affinity.

• debug by pushing the range σ to extremes (wide → Gaussian, narrow → no smoothing); set width by the noise level

• **non-local means**: drop "nearby" — average pixels whose *surrounding patch* matches, anywhere in the image (a bilateral filter in patch space)

• **BM3D**: group similar patches, filter jointly in a transform domain — the long-time benchmark

• sidebar: wavelet / pyramid **coring** — zero the small (noisy) detail coefficients, keep the large (edge) ones — see pyramids

• sidebar: **learned denoisers** (U-Net) beat BM3D; shade into generative priors; **diffusion models** are trained denoisers — see ML

• eyes have coarser acuity for color than brightness → chrominance noise is uglier and safe to smooth hard

• denoise in **YUV**: gentle on luminance, aggressive (large radius) on chrominance — kills color blotches with no visible cost; ties to perception

• you need the noise level to denoise well: **calibrate** it (shoot a flat field at each ISO → a per-ISO noise model), or **estimate from the image** (the std of flat patches / a high-frequency residual)

• there's a floor: **Levin et al.** bound how much *any* denoiser can recover — once noise swamps the local structure, detail is unrecoverable; stronger **priors** (and more photons) are the only way past it

• **side bar (ethics) — when denoising becomes beautification**: pushed too far, denoising (especially on **skin**) erases **texture** (pores, fine lines) → the **"plastic" / wax-figure** look; skin-smoothing, blemish-removal and **beautification** are the *same operation* dialled up (suppress high-freq variation in detected skin). Phones often apply it **by default**, non-consensually — silently imposing a flawless-skin standard, making the photo unfaithful to the person, and affecting **self-image** (esp. young people). Where does a denoiser stop cleaning an image and start editing a face? → ties to ethics ([[Human factors and the art of photography]])

• **quad-Bayer and Nonacell sensors** (2×2 or 3×3 same-color groups — see Sensors) require a **CFA-conversion front-end** step before the normal demosaicking pipeline:

• **full-resolution mode**: a **remosaic** maps each 2×2 (or 3×3) group back to a single-sample-per-site standard Bayer mosaic (e.g. picks the top-left sub-pixel of each group); standard demosaicking then runs as normal

• **binning / low-light mode**: each group is **summed or averaged** into one pixel, producing a quarter- (or ninth-) resolution standard Bayer image; demosaicking follows on that smaller, less noisy mosaic

• the key point: **normal Bayer demosaicking algorithms receive a standard mosaic** either way — the remosaic / binning stage is a CFA-format adapter that sits ahead of it

• → cross-ref Sensors quad-Bayer bullet ([[02 Fundamentals#Sensors photosites CCD vs CMOS]])

• one color per photosite (CCD / CMOS); the **RGGB** pattern; twice as many greens (luminance acuity) [Bayer mosaic figure]

• RAW files: the mosaic **before** demosaicking (linear)

• treat the 3 channels as sparse images, interpolate each (= upsampling); missing green = average of 4 measured neighbours

• works, but artifacts: **zippering** and **color fringing** at edges [demosaick before/after figure]

• **debugging / hand-checkable inputs**: a **constant** image must come back unchanged (nothing to interpolate); a **rectangle** on a flat field makes the **zipper / fringing** appear right at its edges — the minimal inputs that expose demosaicking errors

• a skipped pixel's 4 neighbours straddle the edge → averaging mixes the two sides → values oscillate (the zipper); root cause = **averaging across the edge**

• interpolate along edges, not across them: pick the axis whose two neighbours agree most (smaller |up−down| vs |left−right|)

• bets on a piecewise-smooth world; collapses the zipper

• 💡 **Big lesson:** **let the data pick the direction** — adapt the operation to local structure (precursor to bilateral / affinity)

• R/B are sparser (¼ sampled); harder edge direction

• even perfect per-channel interpolation gives **color fringing**: the 3 channels are sampled at different locations, so their edges don't line up (orange one side, cyan the other) — a between-channel problem

• color channels are highly correlated: **color (R−G, B−G) varies slowly**, brightness has the sharp edges

• recipe: reconstruct green first (edge-directed), form R−G / B−G at measured pixels, interpolate those *naively* (safe — smooth), add green back

• 💡 **Big lesson:** **luminance carries detail, chrominance is smooth** — green ≈ luminance (interpolate carefully), differences ≈ chrominance (interpolate crudely)

• sidebar: difference vs ratio (constant-hue); divide-by-near-zero risk

• still not perfect: fine color texture → **maze / labyrinth** artifacts

• the **OLPF / anti-aliasing filter**: a birefringent layer that slightly blurs before sampling (low-pass before sampling, from the aliasing chapter) — trades sharpness for fewer color artifacts; weakened / dropped as demosaicking improved (computation displacing optics)

• ⚠️ **increasingly omitted** on modern cameras: **higher pixel counts** make aliasing / moiré less visible, and buyers want **maximum sharpness** — a deliberate **sharpness-vs-moiré trade** (drop the filter, accept occasional moiré)

• RAW is noisy; demosaicking and **denoising** are entangled → do them **jointly** (better than in sequence)

• learned joint denoise+demosaick networks (Gharbi et al. 2016; Adobe Enhance Details) — forward ref to ML

• temporal multiplexing (Prokudin-Gorskii, color wheel) — static scenes only

• spatial multiplexing (the mosaic itself, the retina's cone mosaic), 3-chip prism (3-CCD)

• depth multiplexing (film tripack, Foveon)

• ties back to the Color technology

• one of the *first* ISP stages (after black-level / defective-pixel), on linear RAW, **before** white balance / color matrix / tone / sharpen / gamma — order matters

• **the AE problem**: pick aperture/shutter/ISO so the scene lands at the **middle-grey** (~18%) target; the catch is *which* pixels are representative

• **average / center-weighted** metering — average the frame (fooled by bright skies → silhouettes, snow → grey); center-weighting assumes a centered subject

• **matrix / evaluative / pattern** metering — split into **zones**, combine brightness + contrast + focus point (+ color + lens distance = Nikon's **3D Color Matrix**); historically a lookup against a database of reference scenes; the default today

• **spot** metering — meter one small zone; the photographer chooses the anchored tone (Adams's Zone System)

• **learned** metering — a small network (often + semantic segmentation: sky/face/backlight) predicts the photographer's exposure; phones choose exposure jointly with burst/HDR (deliberately under-expose, recover shadows by stacking → Multiple exposure), leaning on the affine+truncated noise model

• 💡 same arc as AWB below: one hand-coded statistic → multi-zone pattern model → learned estimator, all guessing intent from pixels

• spectral perspective

• Von Kries

• **white balance is a per-channel multiply — but only correct in *linear light***: it scales each channel by a gain (Von Kries), and a scaling **commutes with a *linear* map but not a non-linear one**. Apply the same gains to **gamma- / tone-curve-encoded (non-linear)** values and you get the *wrong* answer — the channel gains then **shift hue and change apparent saturation**, not just neutralize the cast. So white balance belongs **early, in the linear pipeline**, alongside exposure (also a multiply-in-linear-light). [fig-wb-linear-vs-nonlinear — the same white-balance gains applied in linear light vs. on gamma-encoded values: the hue/saturation shift] (→ [[#Linear vs gamma vs. log encoding]])

• **white balance from color temperature and tint (the raw-converter slider model)** — the two sliders just pick a **target illuminant white point**, and the diagonal gains follow:

1. **Temperature** $T$ (Kelvin) selects a point on the **Planckian (blackbody) locus** — its chromaticity $(x_c(T), y_c(T))$ from a standard closed-form fit (Kim et al. 2002): warm/low-K → orange, cool/high-K → blue.

2. **Tint** nudges that point **perpendicular to the locus**, along the **green↔magenta** axis (a signed $\Delta uv$ in CIE-1960 $uv$) — correcting casts a pure blackbody can't, e.g. fluorescent green.

3. Convert the white point to tristimulus: $X = x/y,\; Y = 1,\; Z = (1-x-y)/y$, then to the working/sensor space $\mathbf{W} = M_{XYZ\to RGB}\,(X,1,Z)$.

4. Set the **Von Kries gains** $k_c = G_w / W_c$ (normalize to green) so a surface under that illuminant reads **neutral**.

So "$T=5500$ K, tint $0$" and "$(k_R,k_G,k_B)$" are **two faces of the same diagonal correction** — the sliders are a human-friendly parameterization of the per-channel gains. (M lives in [[#Reproducing color]]; *where* the multiply must happen is [[#Linear vs gamma vs. log encoding]].)

• old style: grey assumption, bright pixels

• 3D matrix regression style

• modern ML solutions

• **skin tones** — skin is a **memory colour** cameras are tuned around (face detection feeds the meter + AWB so the *person* lands right); but that tuning has a **history of bias** (the **Shirley card** calibrated film/AE/AWB to light skin, rendering darker skin poorly). Obligation: design & test AE/WB across the full range of skin tones → ethics in [[Ethics of computational photography]]

• multiple / mixed illuminants: one global per-channel gain can't fix a scene lit by two light colors (forward ref: multi-light WB, Hsu et al.)

• metamerism failure: WB can't undo a bad illuminant spectrum (narrow-band fluorescent / cheap LED)

• Color Rendering Index (CRI): how faithfully a source renders colors vs a reference illuminant

uncompressed/raw → lossless (PNG, TIFF) → lossy (JPEG)

lossless = exploit statistical redundancy; lossy = also discard perceptually unimportant info (ties to perception chapter)

• stuff like name, image size

• capture settings (ISO, shutter, aperture, focal length), GPS

• orientation flag (the infamous "why is my image sideways" bug)

• ICC color profile (ties to color management)

• typical-fields **catalog → appendix [[Common EXIF fields]]**; the practical caveats (coverage incomplete & inconsistent; ISO / shutter approximate & not radiometric) are introduced in [[#Image representation]]

-Fun with EXIF: my kids growing up, photos from X years ago

• we use PNG because it's easy to read

• mention alphas in passing to warn people (because they need png with no alpha)

Central idea : leverage human perception (cite early chapter) to adjust the number of bits to what matters. Remind people of CSF and different color sensitivity

• **framing**: JPEG is a perfect illustration that **perceptual criteria can be made quantitative and objective** — "what the eye won't miss" becomes a concrete **quantization table** and bit allocation rather than hand-waving; perception turned into numbers you can compute, tune (the quality knob), and measure

• the **quality knob** = a multiplier on the quantization table; show a **quality sweep** of the same photo (q ≈ 85 → 5) with the resulting file size under each — degradation (softening → blocking → color bleed) grows as the file shrinks [JPEG quality-levels figure]

#### Stage 1: opponent color

Opponent color space

#### Stage 2: chroma subsampling

chroma subsampling (4:2:0 / 4:2:2): coarser chroma than luma because color CSF is lower

#### Stage 3: the 8×8 DCT

DCT

#### Stage 4: quantization — the lossy step

quantization table

#### Stage 5: entropy coding

classic entropy coding, with smart block order

#### Limitations and artifacts

limitation: 8-bit only (motivates other formats); artifacts: blocking, ringing

usually before demosaicking

linear

proprietary vs DNG

• **"RAW" is not always purely raw**: many cameras pre-cook it to some degree — black-level subtraction, defective/hot-pixel correction, sometimes lens-shading/vignetting correction or even mild denoising; some "RAW" is lossy-compressed or non-linearly encoded. So RAW is a *spectrum* of how cooked it is, not literally the untouched sensor readout.

• tools/libraries to read RAW: **dcraw** (the classic), **LibRaw** (C++, dcraw's successor) and its Python wrapper **rawpy**; open-source developers **darktable** / **RawTherapee**; **Adobe DNG Converter / DNG SDK**; **exiftool** for metadata; commercial **Lightroom / Capture One**.

quickly in passing

EXR (OpenEXR), Radiance .hdr, 16-bit TIFF — needed to store HDR output (see HDR chapter)

HEIC/HEIF (Apple default now, 10-bit / HDR-capable), WebP, AVIF, JPEG XL

• **the idea**: replace the hand-designed transform + quantizer + entropy coder of JPEG with a **trained autoencoder** — an **analysis** network maps the image to a latent, the latent is quantized and entropy-coded under a **learned prior**, a **synthesis** network reconstructs. The whole pipeline is optimized **end-to-end** for a **rate–distortion** objective (bits vs. a distortion/perceptual loss), so the codec *learns* what to throw away.

• **why it matters**: at low bitrates learned codecs beat JPEG/even AVIF on perceptual metrics, and they can optimize for **perceptual / GAN** losses rather than MSE (sharper, more natural reconstructions at the same rate). They also generalize: **video**, **depth**, and **scientific** data, and the latents double as features for downstream tasks.

• **the catches**: heavy compute (a neural net to *decode*, not just encode), **non-determinism / version-coupling** across encoder–decoder, and no universal standard yet — though **JPEG AI** (ISO/IEC) is the first standardized learned image codec. Ties to [[Deep learning]] (autoencoders, entropy models) and to generative priors in [[Single-image computational photography]] (super-resolution, inpainting are the same rate–distortion-with-a-prior idea pushed to extreme compression).

• photoshop, layers

• mention alpha again in passing.

• the **image signal processor** runs a fixed **pipeline** from RAW to JPEG; the **order matters** — mostly **linear light** until the end [ISP block-diagram figure]

• **3A** (auto-exposure, auto-white-balance, auto-focus — AF in Optics); **black-level** subtraction; **hot / defective-pixel** correction

• **demosaick** (→ ch) · **denoise** (→ ch) · optional **lens corrections** (vignetting, CA, distortion)

• **white balance** + **color matrix** (sensor → standard RGB) · **tone mapping / curves** (incl. local) · **sharpening**

• **gamma encoding** · **beautification curves** · **JPEG** compression (→ ch)

• advanced: **aberration correction**, computational stages (HDR+, night mode) — later parts

• the throughline (Big lesson): do the physics (WB, demosaick, denoise) in **linear**, **encode (gamma) near the end**

• e.g. **Adobe Lightroom**: keep the **RAW** untouched and store edits as **parameters** (a recipe — exposure, contrast, curves, masks), recomputed on the fly

• benefits: **reversible, re-editable, resolution-independent**; needs a **fast (GPU) pipeline** to recompute interactively

• the parametric pipeline ≈ a programmable ISP exposed to the user

• **the setup**: a sharp scene $x$, a **known** blur kernel $k$ (camera-shake path, defocus disk, lens PSF, motion streak), and the photo we actually got, $y = k * x$. We are **non-blind** here — the kernel is *given* (measured, or recovered earlier); recovering the kernel *too* is **blind** deblurring, deferred to [[Advanced inverse problems]].

• **the one idea**: blur is a **linear, shift-invariant** operation, so it is a **matrix** $A$ acting on the image: $y = Ax$. Deblurring is therefore *inversion* — undo a matrix multiply. Everything below is what "undo" actually means when $A$ is a million-by-million matrix you must never write down. [deblur-linear-inversion figure]

• **three views of the same operator** (the figure ties them together): (i) a **convolution** $y = k * x$ — the intuitive, local view; (ii) a **giant matrix** $A$ — huge ($N\times N$ for $N = W\cdot H$ pixels), but **sparse** (each output touches only the kernel's support) and **structured** (every row is the same kernel, shifted — a *Toeplitz / circulant* matrix); (iii) a **map of vectors** $x \mapsto Ax$ in the image vector space. Same object; we pick whichever view makes the next step easy.

• **why not literal inversion?** Two reasons, both fatal: (1) $A$ is astronomically large — for a 12-megapixel image $A$ is $12\text{M}\times 12\text{M}$; you can't store it, let alone invert it. (2) Even conceptually, $A^{-1}$ is **ill-conditioned** — blur kills high frequencies, so the inverse must *amplify* them, and with them any noise (the deblur preview from the last chapter: naive inverse → exploding noise). We sidestep both: never form $A$ (this chapter), and tame the conditioning with a **prior** ([[Advanced inverse problems]]). [fig-deblur-preview — reuse]

• 💡 **Big lesson:** **diagonalize when you can.** A general blur matrix is a tangled mess to invert, but **Fourier diagonalizes convolution** ([[Linearity, Fourier, Aliasing and deblurring]]): in the frequency basis $A$ becomes a *diagonal* operator — multiply each frequency by $\hat k(\omega)$ — so its inverse is just **per-frequency division** $\hat x(\omega) = \hat y(\omega)/\hat k(\omega)$. That one fact is both the cleanest deblur (the FFT solver, below) and the cleanest explanation of *why* deblurring is hard: where $\hat k(\omega) \approx 0$ (a frequency the blur erased), dividing blows up. *(First appearance: [[Linearity, Fourier, Aliasing and deblurring]]; recurs here — full callback box.)*

• **refresher (images as vectors)**: stack every pixel (and channel) of an image into one long **column vector** $x \in \mathbb{R}^N$, $N = W\cdot H\cdot C$ — exactly how the image already lives in memory (C++ planar $idx = c\,WH + y\,W + x$; NumPy `(H, W, C)` flattened). This **serialization** is mostly *conceptual*: it lets us speak the language of linear algebra (and call its libraries / solvers) without ever materializing the vector or the matrix. [image-as-vector figure]

• **what the vector view buys us**: addition, scaling, the **dot product** $\langle a, b\rangle = \sum_i a_i b_i$ (= correlation / projection), norms $\|r\|^2 = \langle r, r\rangle$ — and the fact that **every linear image operation is now a matrix**: exposure (a diagonal $A$), convolution (a Toeplitz/circulant $A$), downsampling, gradient. One frame, many operations.

• ⚠️ **notation overload (call it out explicitly)**: in linear algebra the unknown image is universally written $x$ — clashing with $x$ = the horizontal pixel coordinate. We keep both (they're standard) and **disambiguate by context**: $x$-the-vector is the whole serialized image; $x$-the-coordinate indexes a pixel. The measurement is $y$ (also overloaded with the row coordinate). *Queue for [[Notations]]:* register $x$ = image-as-vector, $y$ = measurement vector, $A$ = forward (system) operator/matrix, $k$ / $\tilde k$ = blur kernel / flipped kernel, $r = Ax - y$ = residual, $L$ = regularizer (Laplacian) operator.

• **the reframe**: don't ask "solve $Ax = y$ exactly" (with noise there may be **no** exact solution, and the inverse is unstable). Ask instead: **which sharp image $x$, once blurred, best reproduces the photo?** That is **regression** — fit $x$ to the data by minimizing squared error: $\hat x = \arg\min_x \tfrac12\|Ax - y\|^2$. Deblur, denoise, demosaic, HDR-merge, white-balance all share this skeleton; only $A$ changes. [fig-least-squares — reuse: points / residuals / fit, here with $A$ = blur]

• **why squared error**: it makes the problem **linear** (the gradient is linear in $x$), it is the maximum-likelihood fit under Gaussian noise (ties to Refreshers → Probability), and it has a clean closed form. Its blindness to outliers and its need for a prior come later ([[Advanced inverse problems]]).

• **the normal equations**: setting the gradient to zero, $\nabla f(x) = A^\top(Ax - y) = 0$, gives $\boxed{A^\top A\,x = A^\top y}$ — the **normal equations**, the least-squares analogue of "$Ax = y$." $A^\top A$ is **symmetric positive (semi-)definite**, which is exactly the structure the iterative solvers below exploit.

• **what $A^\top$ *is*, concretely**: the **transpose of a convolution by $k$ is a convolution by the flipped kernel** $\tilde k(\mathbf{u}) = k(-\mathbf{u})$ — i.e. correlation. So $A x = k * x$ (blur) and $A^\top y = \tilde k * y$ (the "back-projected" blur). This is the crucial hook: **both $A$ and $A^\top$ are just convolutions**, so we can apply them without ever building a matrix.

• **deferred (scope boundary)**: noiseless least squares over-fits noise — the cure is a **regularizer / prior**, $\min_x \tfrac12\|Ax - y\|^2 + \lambda R(x)$, turning the normal equations into $(A^\top A + \lambda L)\,x = A^\top y$. We name it here and **defer the why/how to [[Advanced inverse problems]]** (Tikhonov, total-variation, Wiener) and the *learned* prior to [[Machine learning]]. The solver machinery below is unchanged — only the operator picks up a $+\lambda L$.

• **the central trick of the chapter**: the normal-equation matrix $A^\top A$ is $N\times N$ and we will **never build it**. Every iterative solver below needs **only one primitive**: given a vector $v$, return $A^\top A\,v$ — computed as **two convolutions**, $A^\top(A v) = \tilde k * (k * v)$. *Matrix-free.* `apply_blur(v) = conv(v, k)`, `apply_blur_T(v) = conv(v, flip(k))`; the solver calls these and nothing else.

• **gradient descent**: walk downhill on $f(x)=\tfrac12\|Ax-y\|^2$. The gradient is $\nabla f(x) = A^\top(Ax - y)$ — **one forward blur, one residual, one transposed blur** per step — then $x_{t+1} = x_t - \eta\,\nabla f(x_t)$. Correct and trivial to code, but **slow**: the least-squares bowl is **elongated** (badly conditioned, because blur squashes high frequencies), so plain gradient descent **zig-zags** down the valley. [fig-gradient-descent — reuse: steps down a quadratic bowl]

• *pseudocode:* a matrix-free gradient-descent loop (init $x{=}0$; iterate: residual $A x{-}y$, gradient $A^\top$ of it, step by $-\eta\nabla$) using only the two convolutions.

• **conjugate gradient (CG)** — the workhorse: because $A^\top A$ is **symmetric positive-definite**, CG solves the normal equations in far fewer iterations by choosing **conjugate** (non-interfering) search directions instead of greedily steepest ones — it never undoes earlier progress, so it cuts straight down the elongated valley. Like gradient descent it touches $A$ only through **apply-operator** calls (one $A$ + one $A^\top$ per iteration); unlike it, it converges in tens, not thousands, of steps. Cite Shewchuk's *"…Without the Agonizing Pain"* for the intuition. [deblur-cg-vs-gd figure — CG's conjugate path vs GD's zig-zag on the same bowl]

• **preconditioning**: CG is fast when the bowl is **round** (well-conditioned). A **preconditioner** $M \approx (A^\top A)^{-1}$ that is cheap to apply (e.g. a diagonal, or an FFT-based approximate inverse) **re-rounds** the bowl and slashes the iteration count — preconditioned CG is the practical default.

• **the situation**: $A^\top A$ is **huge but sparse and structured**, so in practice we use **iterative / multiscale** solvers — gradient descent and conjugate gradient (above), plus **multigrid** (coarse-to-fine on a pyramid) and the **FFT solver** (when the operator diagonalizes in Fourier) — never a direct inverse. These are the shared workhorses of the whole part, so they get their **own chapter next** ([[#Efficient solvers]]): which solver wins when, and why each is matrix-free.

• **the payoff (why these two chapters are the foundation of the part)**: this **same solver machinery** — set up $y = Ax$, minimize $\|Ax - y\|^2$ (+ a prior later), solve matrix-free with CG / multigrid / FFT — recurs across **gradient-domain HDR, [[Poisson image editing|Poisson cloning]], colorization, matting**, and most optimization-based edits in this part. Learn it once, here and in [[#Efficient solvers]]; every later chapter is a new $A$ and a new prior on the same spine.

• **the situation**: $A^\top A$ is **huge but sparse and structured**, so we use **iterative / multiscale** solvers, never a direct inverse. Everything below touches $A$ only through the matrix-free pair `apply_blur(v) = conv(v, k)` and `apply_blur_T(v) = conv(v, flip(k))`. Three (plus one) tools, ordered by when each wins.

• **gradient descent and CG** (developed in the previous chapter) are the starting point: GD walks downhill on $\tfrac12\|Ax-y\|^2$ but **zig-zags** on the elongated, ill-conditioned least-squares bowl; **conjugate gradient** chooses non-interfering directions and converges in tens, not thousands, of steps. Both use only apply-operator calls. [deblur-cg-vs-gd figure — CG's conjugate path vs GD's zig-zag]

• *pseudocode:* the operator-only CG loop (residual, conjugate search direction, exact step $\alpha$, conjugacy update $\beta$) — one $A^\top A$ matvec = two convolutions per iteration, no matrix formed.

• **conjugate gradient + preconditioner** — the general-purpose default; works for *any* $A$ you can apply (and apply-transpose), even when the boundaries or the operator aren't shift-invariant. A **preconditioner** $M \approx (A^\top A)^{-1}$ that is cheap to apply re-rounds the bowl and slashes the iteration count — preconditioned CG is the practical default.

• **multigrid (coarse-to-fine)**: low-frequency error is what iterative solvers kill *slowest*, and it is exactly what a **coarse grid** represents cheaply. Solve on a coarse image, **prolong** (upsample) the correction, refine on the fine image — a **V-cycle** down and up an image **pyramid** — and convergence becomes near-linear in pixel count. Directly reuses the Gaussian/Laplacian-pyramid machinery of [[Linear pyramids and wavelets]] (reduce / expand). [deblur-multigrid-vcycle figure — V-cycle down and up the pyramid]

• **FFT solver**: when the domain is **rectangular with periodic (or Neumann) boundaries**, the blur (and a Laplacian regularizer) is **circulant → diagonal in Fourier** (L5). Then there is **no iteration at all**: transform, **divide per frequency** $\hat x(\omega) = \hat y(\omega)\,\overline{\hat k(\omega)} \,/\, \big(|\hat k(\omega)|^2 + \lambda\,|\hat L(\omega)|^2\big)$, inverse-transform. Two FFTs and a pointwise divide — the fastest solve when its boundary assumption holds; ties to [[Linearity, Fourier, Aliasing and deblurring]]. [deblur-fft-solver figure — deblur as per-frequency division]

• **picking one**: **CG + preconditioner** for any operator you can apply (the safe default); **multigrid** when low-frequency error dominates and you have a pyramid; **FFT** when the operator is shift-invariant with nice boundaries (the fastest, but most restrictive).

• **the payoff (why this chapter is the foundation of the part)**: this **same solver machinery** — set up $y = Ax$, minimize $\|Ax - y\|^2$ (+ a prior later), solve matrix-free with CG / multigrid / FFT — recurs across **gradient-domain HDR, [[Poisson image editing|Poisson cloning]], colorization, matting**, and most optimization-based edits in this part. Learn it once here; every later chapter is a new $A$ and a new prior on the same spine.

• **the move.** Every task in this part so far had a **hand-designed operator** — an inverse filter, a smoothness prior, a dark-channel heuristic. Replace it with a function $f_\theta$ (a net) **fit to example data**: collect pairs (degraded input, desired output), minimize a loss, deploy the learned map. [fig-learned-vs-handdesigned]

• **same skeleton, data-driven prior.** Recall the inverse-problem template $\hat I=\arg\min_I \lVert AI-b\rVert^2+\lambda R(I)$ ([[Optimization and regression]]). A learned method keeps **data-fit + prior**; it just replaces the hand-tuned $R$ (or the whole solver) with something **learned from a dataset** — so the network *is* the prior.

• 💡 **Big lesson:** L8 — *a learned operator swaps a hand-designed prior for one learned from data.* Same data-fit + prior skeleton as the inverse-problem chapters; the prior is now fit to examples, not hand-tuned. Cost: data, compute, and **hallucinated** detail. (first appearance — full box here; the ML refresher carries the one-line callback) [[Big Lessons#L8]]

• **why now (and why it works).** Big datasets + GPUs + CNN/U-Net/transformer building blocks make fitting these maps practical; intuition over derivation (nets are the Refreshers' job). One-line nod to **the Bitter Lesson** (Sutton): general methods that scale with data/compute tend to win over hand-engineered ones — but flag the costs and failure modes rather than cheerlead.

• **scope.** This chapter sets the **framing** and the **data** that powers learned operators; the **operator zoo** itself — low-level pixel-to-pixel, mid/high-level predictors, generative translation, and learned metrics — is the next chapter, [[Deep learning]]. Architectures and training: the Refreshers throughout.

• refs: Sutton 2019 (The Bitter Lesson); Torralba/Isola/Freeman Part III; Szeliski ch. 5.

The recurring lesson of this chapter: **the data is the method.** A learned operator is only as good as the pairs it trains on, and good pairs are often *manufactured*.

• **generating synthetic data.** Start from clean targets and **simulate the degradation** to make (input, target) pairs at scale — the cheap way to get supervised data when ground truth is otherwise unobtainable (you can't photograph the same scene "clean" and "noisy" perfectly registered). [fig-synthetic-data-pipeline]

• **realistic noise models (the crux).** A net trained on *additive Gaussian* noise fails on real photos — real sensor noise is **signal-dependent (Poisson shot + read)**, post-ISP, and demosaicked. Match the **real noise model** (cross-ref the shot-noise / Poisson discussion, Refreshers & Book 2) or the learned denoiser won't transfer. This is why Gharbi mines hard cases and Real-ESRGAN models a complex degradation chain.

• **reverse-engineering the camera pipeline (ISP).** To synthesize realistic raw/finished pairs you must **invert/replay the ISP** — black level, white balance, demosaic, tone curve, compression — so synthetic data looks like the camera's actual output. Ties to FlexISP / the camera-pipeline material.

• **standard datasets & benchmarks.** Name the workhorses and *what each measures*: ImageNet (pretraining/features), DIV2K (super-res), SIDD/DND (real-world denoising), MIT-Adobe FiveK (retouching), KITTI/NYU (depth), COCO (detection/segmentation). The benchmark *defines the task*; its biases become the model's biases. (Full index + links: [[Appendices#Datasets]].)

• **a note on self-supervision (mention, don't expand).** Two spots here sidestep the data problem rather than brute-forcing it: (1) **self-supervised denoising** (Noise2Noise / Noise2Void) trains *without clean targets at all* — a direct answer to "we can't photograph the clean version"; (2) **pretrained backbones** behind depth/detection/perceptual metrics often come from self-/weakly-supervised pretraining on unlabeled images. Both matter; neither needs its own section — deep treatment belongs to the ML appendix / advanced books.

• refs: Gharbi et al. 2016; Wang et al. 2021 (Real-ESRGAN degradation model); Heide et al. 2014 (FlexISP); Bychkovsky et al. 2011 (FiveK); Lehtinen et al. 2018 (Noise2Noise) — **to download**; queue dataset citations (DIV2K, SIDD, NYU-Depth) — **to download.**

The bread-and-butter: an image-to-image net (typically a **U-Net** / encoder–decoder) maps a degraded or partial image to a restored one. Each task below is the *classical* chapter's task, now learned. [fig-learned-task-zoo]

• **denoising.** Train on (noisy, clean) pairs → a learned denoiser that beats hand-tuned NLM/bilateral, *if* the training noise matches the real sensor noise (→ the data story, [[Machine learning]]). Forward-ref: the denoiser as a reusable prior ([[Super-resolution and image priors#Denoising as a universal prior — Plug-and-Play and RED|Denoising as a universal prior]] — PnP/RED) and diffusion as iterated denoising ([[Generative AI and diffusion]]).

• **demosaicking — and joint demosaick+denoise.** Gharbi et al.: a single net learns to go **mosaic → full RGB**, jointly with denoising, trained on hard cases mined from data — outperforming the hand-built interpolators of Book 2. The headline case that "learn the operator end-to-end" beats a decades-tuned pipeline stage. [fig-learned-task-zoo: mosaic→RGB cell]

• refs: Gharbi et al. 2016 (deep joint demosaicking & denoising).

• **super-resolution.** Pose low-res → high-res as a learned map. Lineage from **Freeman et al.** (example-based / patch-dictionary super-res — the pre-deep ancestor) to modern nets: **SwinIR** (transformer restoration backbone) and **Real-ESRGAN** (GAN-trained on a *realistic* synthetic degradation model for in-the-wild photos). Note: the deep treatment of SR + priors may live in [[Super-resolution and image priors]]; here it's one entry in the zoo with a forward-ref.

• refs: Freeman et al. 2002 (example-based SR); Liang et al. 2021 (SwinIR); Wang et al. 2021 (Real-ESRGAN).

• **colorization — classical vs learned, side by side.** Two paradigms: **Levin et al. 2004** propagates a few user **scribbles** by optimization (edge-preserving — "same color where intensity is similar", an *affinity* prior, cross-ref L4); **Zhang, Isola & Efros 2016** learns **fully automatic** colorization from a huge corpus, posed as classification over quantized colors to fight desaturation. Contrast: hand-designed propagation vs learned-from-data plausibility. [fig-colorization-classical-vs-learned]

• refs: Levin, Lischinski & Weiss 2004 (colorization by optimization); Zhang, Isola & Efros 2016 (colorful image colorization).

• **monocular depth.** Estimate depth from a *single* RGB image — ill-posed, so the net learns a strong **scene prior**. **MiDaS**: trained across many datasets via a **scale/shift-invariant** loss to mix incompatible depth sources; **Depth Anything**: scale further with massive (pseudo-labelled) data → robust zero-shot depth. Note it's *relative* depth, and forward-ref geometric/dual-pixel depth (Books on geometry / multi-image). [fig-depth-anything]

• refs: Ranftl et al. 2019 (MiDaS); Yang et al. 2024 (Depth Anything).

• **learned exposure / retouch (tone & color adjustment).** Learn a photographer's **global/local adjustment** from before/after pairs. **Bychkovsky et al. (MIT-Adobe FiveK)**: the dataset of expert retouches that made this learnable. **HDRnet** (Gharbi 2017): predict the *coefficients of a bilateral-grid affine transform* — fast, edge-aware, runs on a phone (cross-ref the learned bilateral grid in edge-preserving). **Exposure** (Yuanming Hu 2018): learn a sequence of *interpretable* retouching operations (curves, white balance) rather than raw pixels — keep as the "learn the *operations*, not the pixels" variant.

• refs: Bychkovsky et al. 2011 (FiveK); Gharbi et al. 2017 (HDRnet); Hu et al. 2018 (Exposure).

Not pixel-to-pixel restoration but **prediction of semantic structure** from a single image — the inputs to retouching, autofocus, framing, and selection.

• **sky / face detection.** The classic "where is the sky / where are the faces" detectors that drive auto-enhance and metering; the pre-deep (Viola–Jones-era) lineage → modern detectors. Keep brief; they're enabling, not the focus.

• **face landmarks & pose.** Predict facial keypoints + head pose — feeds portrait retouching, relighting, AR, and the *forensics* angle (consistency checks against deepfakes — cross-ref ethics). (Resolves the "photo bio here?" stub: **cut** — photo-bio/EXIF belongs in file formats, not here.)

• **saliency / gaze prediction (what draws the eye).** ML supplies the *estimator* (pre-deep Itti–Koch → learned saliency trained on eye-tracking data), but the model and its uses (auto-crop/framing, retargeting, attention-aware compression, quality/aesthetics) now live with the other perceptual models in **[[Human factors#Computational models of perception]]** — moved there so the *human-factors* framing owns it. Mentioned here only as one more learned predictor.

• refs: queue — Viola & Jones 2004 (face detection); learned-saliency ref (SALICON / DeepGaze) — **missing from `refs/`, to download.** (Saliency refs proper: see [[Human factors#Computational models of perception]].)

When the target isn't a unique "correct" image but a **plausible** one, shift from regression to **generative** modelling.

• **why generative.** Many single-image tasks are **one-to-many** (colorization, super-res, inpainting) — averaging plausible outputs (L2) gives blur; a generative model samples a *sharp, plausible* answer instead. Motivates the next two.

• **GANs (adversarial training).** A generator vs a discriminator that learns the "real-image" prior → sharp, plausible detail (the engine behind Real-ESRGAN's realism). One-paragraph intuition; depth deferred to [[Generative AI and diffusion]].

• **conditional / paired & unpaired translation.** **Pix2Pix** (Isola et al.): paired image-to-image translation — edges→photo, label-map→scene, day→night — a generic "learn the map between two image domains" tool. **CycleGAN**: the **unpaired** version (cycle-consistency) when you lack aligned pairs (horses↔zebras, summer↔winter). [fig-gan-pix2pix]

• **the pre-deep ancestor.** Mention **Image Analogies** (Hertzmann et al. 2001): "A:A′ :: B:B′" learned a filter/style from a *single* example pair by patch matching — the same translation idea before deep nets (ties to texture synthesis / patch-match).

• **forward-ref.** Diffusion models now dominate generative image-to-image (text-to-image, editing) — full treatment in [[Generative AI and diffusion]]; here just the placement (GAN → diffusion lineage).

• refs: queue — Isola et al. 2017 (Pix2Pix), Zhu et al. 2017 (CycleGAN), Goodfellow et al. 2014 (GANs), Hertzmann et al. 2001 (Image Analogies) — **missing from `refs/`, to download.** Forward-ref: Rombach et al. 2022, Brooks et al. 2023 (in the diffusion chapter).

How do we *measure* whether a restored/generated image is good — and how do we *train* for it? The classical metrics break exactly where learned methods shine, and the metric you optimize decides the sharpness.

• **back-ref: why L2 / PSNR / SSIM / VDP fall short.** Introduced in Book 2: per-pixel L2 / **PSNR**, structural **SSIM**, and perceptual **VDP** all compare *low-level* statistics. They are **blind to texture and high-level similarity** — a slightly shifted but perceptually-identical texture scores terribly; a blurry mean scores *well*. So they reward exactly the wrong thing for generative restoration. [fig-perceptual-metric]

• **LPIPS — deep feature distance.** Compare two images by the distance between their **deep network features** $\ell_{\text{feat}}=\sum_l\lVert\phi_l(\hat I)-\phi_l(I)\rVert^2$, calibrated to human similarity judgments. Tracks perceived similarity far better than L2; it's also usable as a **training loss** (cf. Gatys' feature/perceptual loss, neural style).

• **DreamSim — learned perceptual embedding.** A more recent metric tuned to *mid-level* / **holistic** similarity (layout, pose, semantic content), capturing judgments LPIPS misses. Keep as "the next step beyond LPIPS."

• **from metric to loss — perceptual losses buy sharpness.** Training on per-pixel **MSE/L1** regresses to the **blurry mean**; adding a **perceptual (feature/LPIPS) loss** restores texture and an **adversarial (GAN)** term sharpens onto the real-image manifold. Caveat: perception–distortion trade-off (**L10**) — extra sharpness is partly hallucinated, so keep a fidelity anchor. [fig-perceptual-loss-sharpness — same input restored under L2 vs L1+perceptual vs +adversarial; blurry → crisp]

• **a go-to loss mix** for restoration/super-res/deblur: weighted sum $\mathcal{L}=\lambda_1\lVert\hat I-I\rVert_1 + \lambda_p\,\ell_{\text{LPIPS}} + \lambda_a\,\mathcal{L}_{\text{adv}}$ — **L1** fidelity/colour anchor ($\lambda_1{=}1$), **LPIPS/VGG** perceptual for texture ($\lambda_p{\approx}0.1$–$1$), a **small adversarial** term for crisp highs ($\lambda_a{\approx}0.01$–$0.1$, added last). Track LPIPS not PSNR.

• one-line takeaway: **metrics are themselves learned now** — and which metric you optimize quietly defines what "good" means (and what the model hallucinates toward).

• refs: back-ref Book 2 (PSNR/SSIM/VDP); Gatys, Ecker & Bethge 2016 (feature/perceptual loss); Zhang et al. 2018 (LPIPS) — **to download**; Fu et al. 2023 (DreamSim) — **to download.**

• **the move.** Every prior so far was something you **evaluate**: a penalty $\Phi(x)$ you *add* to a data term ($\min_x \tfrac12\|Ax-y\|^2 + \lambda\,\Phi(x)$), or a denoiser $\mathcal D$ you *plug* into a solver (PnP/RED). A **generative model** learns the distribution of natural images $p(x)$ well enough that you can **draw fresh samples** $x\sim p(x)$ — not score an image, but *produce* one. [fig-genai-evaluate-vs-sample]

• **why sampling changes everything.** With a sampleable prior you get three capabilities the evaluable prior couldn't give: (1) **unconditional generation** — invent a plausible image from nothing; (2) **conditioning** — sample from $p(x\mid c)$ for a text prompt, a sketch, a class, another image (text→image, image→image); (3) **posterior sampling** — condition the prior on a *measurement* $y$ and sample $p(x\mid y)\propto p(y\mid x)\,p(x)$, which is exactly **super-resolution / deblur / inpaint** with the strongest prior we have. The inverse-problem template is unchanged; the prior just learned to *generate*.

• **one-to-many is the point.** Restoration tasks like colorization, super-res, and inpainting are **one-to-many** — many sharp images explain one degraded input. Regression (L2) averages them into a blurry mean (the very failure motivated in [[Deep learning]]); a generative prior **samples one sharp, plausible answer** instead. [fig-posterior-sampling]

• 💡 **Big lesson:** L8 — *a learned operator swaps a hand-designed prior for one learned from data.* The generative prior here is the most extreme case of that swap: the entire distribution of natural images, learned. (Recurrence; first appears in [[Machine learning]].) [[Big Lessons#L8]]

• 💡 **Big lesson:** L10 — *the prior is not optional.* When the measurement destroys information (SR, inpaint, deblur), only a prior selects an answer. A generative model is that prior at full strength — and conditioning it on the measurement is posterior sampling. (Recurrence; first appears in [[Super-resolution and image priors]].) [[Big Lessons#L10]]

• 💡 **Big lesson:** L11 — *a generative model is a sampleable prior over images.* The earlier chapters' prior was something you *evaluate* (a penalty $\Phi$) or *denoise with*; a generative model is a prior you can **draw images from**, $x\sim p(x)$. That leap — from scoring to sampling — is what makes generation, text conditioning, and posterior sampling for inverse problems possible. (First appearance; registered as [[Big Lessons#L11]].)

• **the pre-deep ancestor.** "Sample plausible pixels" predates deep nets: **Efros–Leung texture synthesis** (1999) grew an image pixel-by-pixel by copying from matching neighbourhoods — a non-parametric sample from the patch distribution. Same instinct (draw from $p(x)$), no learned model. Ties to the patch/texture material and to Image Analogies ([[Deep learning]]).

• refs: Efros & Leung 1999 (texture synthesis); Goodfellow et al. 2014; Ho et al. 2020; Torralba/Isola/Freeman Part IX.

• **the two chains.** A **forward** process gradually corrupts an image into pure Gaussian noise: $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$ — at $t=0$ the clean image, at $t=T$ static. A **reverse** process learns to undo one step of that corruption at every noise level. **Sampling** runs the reverse chain from $x_T\sim\mathcal N(0,I)$ down to a clean $x_0$ — an image *emerging from noise*. [fig-diffusion-forward-reverse]

• *pseudocode:* the reverse sampling loop (start from $x_T\sim\mathcal N(0,I)$; for $t$ from $T$ down to $1$: predict the noise $\epsilon_\theta$, denoise a step, reinject a little fresh noise) — the "sample, don't average" walk.

• **what the network learns (denoise everywhere).** Train a single net $\epsilon_\theta(x_t,t)$ to **predict the noise** added at level $t$, by minimizing $\mathbb{E}_{x_0,\epsilon,t}\lVert \epsilon - \epsilon_\theta(x_t,t)\rVert^2$ over clean images, random noise, and random levels. Predicting the noise *is* denoising — the model is a **denoiser at every noise level**.

• **the hinge — the score is a denoiser (Tweedie).** For a Gaussian-noised $x_t$, the MMSE denoiser (posterior mean) is $\hat x_0 = x_t + \sigma_t^2\,\nabla_{x_t}\log p_{\sigma_t}(x_t)$: the **score** $\nabla\log p$ and a **Gaussian denoiser** are the *same object*, one rearranged into the other. So a diffusion model *is* a learned score field / family of denoisers. (First appearance of this identity: *Denoising as a universal prior*, [[Super-resolution and image priors]]; recurs here as the chapter's spine.) [fig-score-is-denoiser]

• **therefore diffusion = PnP/RED at the continuous limit.** The sampling loop — *denoise a little, add a little noise, step down the schedule, repeat* — is exactly the **Plug-and-Play / RED** outer loop ([[Super-resolution and image priors]]), with the prior-step denoiser being the learned score run over a noise-level schedule. Generation is iterated denoising; the book's denoiser-as-prior abstraction and its hottest generative model are one mechanism.

• 💡 **callback to L10/L11:** the not-optional prior (L10) is now a *sampleable* one (L11) — and the machinery is the universal denoiser of the previous chapter, looped.

• **latent diffusion (why it scaled).** Diffusing in **pixels** at high resolution is expensive. **Latent diffusion** (Rombach 2022, Stable Diffusion) first **encodes** the image into a compact latent with an autoencoder, runs the whole diffusion process **in that latent**, then **decodes** — far cheaper, and the step that made open high-res text→image practical. [fig-latent-diffusion]

• refs: Ho et al. 2020 (DDPM); Song et al. 2021 (score SDE / samplers); Rombach et al. 2022 (latent diffusion); back-ref Venkatakrishnan 2013 / Romano 2017 (PnP/RED — the loop diffusion generalizes).

• **text→image.** A frozen **text encoder** (e.g. CLIP / T5) turns the prompt into embeddings that condition the denoiser at every step via **cross-attention** — the model samples from $p(x\mid \text{text})$ instead of $p(x)$. This is the headline capability of Stable Diffusion / DALL·E / Imagen-style systems.

• **text vs images — why two different generative models** (resolving the outline's old *Text vs images* stub): text is **discrete tokens** with a natural left-to-right order and a finite vocabulary, a perfect fit for **autoregressive** language models ($p(x)=\prod_i p(x_i\mid x_{<i})$, exact next-token likelihood); images are **continuous**, million-dimensional, and have **no canonical ordering**, a perfect fit for **diffusion** (add/remove continuous Gaussian noise over *all* pixels in parallel, no ordering). It is not fashion but **discrete-sequential vs continuous-parallel** — and the two modalities meet through a shared **CLIP** embedding, where an LLM-encoded prompt steers the image diffusion model.

• **classifier-free guidance (the knob).** Train the model **with and without** the condition (dropping $c$ sometimes), then at sampling time **extrapolate** along the conditioned-vs-unconditioned difference: $\tilde\epsilon_\theta = \epsilon_\theta(x_t,\varnothing) + w\big(\epsilon_\theta(x_t,c)-\epsilon_\theta(x_t,\varnothing)\big)$. The guidance weight $w$ trades **prompt fidelity vs diversity** — the single most-used quality dial.

• **structural control (ControlNet).** Beyond text, condition on a **spatial hint** — edges, pose, depth, segmentation — so the output obeys a given structure *and* the prompt. **ControlNet** (Zhang 2023) adds a trainable branch that injects the hint without retraining the base model. [fig-conditioning-controlnet]

• **instruction-based editing.** **InstructPix2Pix** (Brooks 2023): condition on *(an input image, a text instruction)* — "make it winter," "turn the car red" — and the model edits in place. The image→image lineage runs from Pix2Pix/CycleGAN ([[Deep learning]]) to diffusion editing here.

• refs: Ho & Salimans 2022 (CFG); Zhang et al. 2023 (ControlNet); Brooks et al. 2023 (InstructPix2Pix); Rombach et al. 2022.

• **the payoff for the rest of the book.** Plug the generative prior back into the inverse-problem template. To solve $y=Ax+n$, **sample the posterior** $p(x\mid y)\propto p(y\mid x)\,p(x)$ by **alternating** a diffusion prior step (denoise toward the manifold of natural images) with a **data-fit** step against the forward model $A$ — a PnP/RED solver whose prior is the strongest learned denoiser we have. Result: state-of-the-art **super-resolution, deblurring, inpainting, colorization**. [fig-posterior-sampling]

• **reconstruction vs hallucination (the honest caveat).** Because the prior now *invents* plausible detail, posterior sampling sits squarely on the **hallucination** side of L10's split: the detail is plausible, not measured. Different samples give different valid answers — useful (uncertainty) and dangerous (invented "evidence"). Always label which is which.

• **where it gets applied.** This abstraction is what the later application parts reach for: **compositing / inpainting** (fill a removed object), **relighting**, **video** (temporally-conditioned generation), and generative **super-resolution**. Here we establish the mechanism; each part applies it.

• refs: back-ref Venkatakrishnan 2013 / Romano 2017 (PnP/RED); Song et al. 2021 (conditional sampling); [[Super-resolution and image priors]] (the prior throughline this closes).

• **GANs.** A generator vs a discriminator that learns the "real-image" boundary → sharp samples in one forward pass (the engine behind Real-ESRGAN's realism, [[Deep learning]]). Fast to sample, historically hard to train (mode collapse); largely overtaken by diffusion for general text→image, still used where one-shot speed matters.

• **VAEs.** Encode→sample-latent→decode; a principled probabilistic model, but samples are typically **blurrier**. Live on inside **latent diffusion** as the autoencoder that defines the latent space.

• **autoregressive / transformer image models.** Generate tokens (pixels or codebook entries) one at a time (PixelRNN/CNN, image transformers, VQ-based models); the same next-token idea as language models, now reborn in multimodal systems.

• one-line placement: **diffusion currently dominates** open image generation; the others matter as alternatives, components, or speed/latency trade-offs.

• refs: Goodfellow et al. 2014 (GANs); Kingma & Welling 2014 (VAE); van den Oord et al. 2016 (PixelCNN — queue ref).

• **hallucination is structural, not a bug.** A sampleable prior *must* invent detail — that is what makes it work and what makes it untrustworthy as evidence. Generated/super-resolved detail is **plausible, not real**; never treat it as measurement (forensics, medical, legal). Ties to L10's reconstruction-vs-hallucination split and to [[Human factors and the art of photography]].

• **bias and representation.** The model can only sample what its **training distribution** contained; skews in the data become skews in the outputs (demographics, styles, defaults). The benchmark/dataset defines the behaviour, as in [[Machine learning]].

• **deepfakes, provenance, consent.** Realistic synthesis enables **deepfakes** and misinformation; mitigations — watermarking, provenance/C2PA, detection (the forensics angle, cross-ref face-landmark consistency checks) — are partial. Training on scraped images raises **consent, copyright, and attribution** questions still unsettled.

• **environmental and access costs.** Training and serving large diffusion models is compute- and energy-intensive; capability concentrates with whoever can pay — flag, don't cheerlead (echoing the Bitter-Lesson caveat in [[Machine learning]]).

• refs: queue — provenance/C2PA, a deepfake-detection survey, a dataset-consent reference — **missing from `refs/`, to download**; cross-ref [[Human factors and the art of photography]] (ethics, forensics, the art-vs-evidence line).

• the **key idea**: the eye cares about **gradients** (local contrast), not absolute values — so edit the **gradient field** $\mathbf{v}$ and reconstruct the image $f$ whose gradients best match it. A field rarely has an exact integral (it is generally not a true gradient — see the conservative-fields sidebar), so it is a **least-squares** reconstruction $\min_f \int \lVert \nabla f - \mathbf{v}\rVert^2$, whose normal equations are the **Poisson equation** $\nabla^2 f = \operatorname{div}\mathbf{v}$ — i.e. exactly the inverse-problem machinery of [[Linear Inverse Problems and Regression]], applied to gradients.

• 💡 **Big lesson:** *the eye cares about gradients (local contrast), not absolute values* — so edit the gradient field and reconstruct. A mismatched absolute level (a paste's lighting/color shift, an HDR scene's enormous range) is invisible once the *gradients* are right; this is the same reason lightness constancy works and the visual system encodes differences (→ register/first appearance: [[Big Lessons#L9]]).

• **seamless cloning** (the headline app, Pérez 2003): to paste a region $\Omega$ without a visible seam, copy the source's **gradients** $\mathbf{v}=\nabla g$, not its pixels, but pin the interior to the destination's **boundary values** $f|_{\partial\Omega}=f^{*}|_{\partial\Omega}$ (a **Dirichlet** condition); solving $\nabla^2 f=\operatorname{div}\mathbf{v}$ fills an interior that flows smoothly out of the destination at the seam, so lighting/color mismatches are absorbed and the paste blends invisibly. `fig-seamless-cloning`

• the discrete **Laplacian** (a convolution kernel — the 5-point stencil) turns $\nabla^2 f = \operatorname{div}\mathbf{v}$ into **one equation per pixel**, $4f_p-\sum_{q\in N(p)}f_q=\sum_{q\in N(p)} v_{pq}$ over the 4-neighbours $N(p)$ — a **large, sparse, structured** linear system. **Boundary conditions** close it: **Dirichlet** (fixed boundary $f|_{\partial\Omega}=f^{*}|_{\partial\Omega}$) for cloning; **Neumann** (zero normal derivative) when integrating a whole image with no enclosing destination.

• pseudocode block: the transparent **Jacobi / Gauss-Seidel** iterate-to-convergence solve — each pixel ← ¼·(sum of 4 neighbours + divergence rhs), swept until settled, boundary held fixed — as the slow baseline the CG / multigrid / FFT solvers accelerate.

• the **Fourier-domain view (one FFT divide)**: gradient and Laplacian are **convolutions** (linear shift-invariant), so on a rectangular domain with periodic/Neumann boundaries they are **diagonal in the Fourier basis** — every frequency decouples and the whole Poisson equation $\nabla^2 f=\operatorname{div}\mathbf{g}$ solves in **one per-frequency division**, $\hat f(\omega)=\widehat{\operatorname{div}\mathbf{g}}(\omega)/\widehat{\nabla^2}(\omega)$ (dividing by the Laplacian's spectrum $-(\omega_x^2+\omega_y^2)$; a **DCT** variant handles Neumann/free boundary conditions cleanly). This is the **same "diagonalize when you can" (L5) FFT solver** as in deblurring — see [[Efficient solvers]] / [[Computational tools]] and [[Linearity, Fourier, Aliasing and deblurring]]. Subtlety: the **DC / mean is undetermined** — the Laplacian's spectrum is **zero at $\omega=0$** (its zero eigenvalue), so gradients pin down everything **but the overall offset**, which must be fixed by a boundary value or a reference. [`fig-poisson-fourier`: the Poisson solve as a per-frequency division — divide by the Laplacian's spectrum $-(\omega_x^2+\omega_y^2)$]

#### The same solver as the regression chapter

• **same solver as the regression chapter**: don't form the matrix — use **conjugate gradient** (with a preconditioner), **multigrid** (coarse-to-fine — ties to [[Linear pyramids and wavelets]]), or an **FFT solver** on a rectangular periodic/Neumann domain (the Laplacian **diagonalizes in Fourier** — ties to [[Linearity, Fourier, Aliasing and deblurring]]). This is the *same* machinery as [[Linear Inverse Problems and Regression]], colorization, and matting.

#### Which space to integrate in: linear vs. gamma vs. log

• **which space to integrate in (linear vs gamma vs log)**: integrate in a space where equal *gradients* mean equal *perceived* contrast. Plain **linear** light over-weights highlights and lets bright regions dominate the seam; **log** (or a perceptual encoding) makes a ratio a fixed gradient, which matches the eye and keeps shadow detail — so for tone/exposure-mismatched clones and gradient-domain HDR we integrate in **log**, then map back. We state the chosen encoding explicitly per edit (the encoding-clarity rule).

• the **membrane** lens on the two seam-hiders: **Poisson** = gradient-domain + pinned boundary → copies *gradients* and **discards the source DC** (re-set by the boundary; adding a constant changes nothing); **Laplacian-pyramid blend** ([[Linear pyramids and wavelets]]; Burt–Adelson spline) = cross-fade *values* band-by-band → **keeps the DC**, feathered by a coarse-wide mask

• discriminator — **paste a constant**: Poisson collapses to the bare **membrane** (Laplace, boundary = destination) and the constant *vanishes* (a line in 1-D, a soap film in 2-D); the pyramid's coarse band **copies the DC**, so a feathered constant *survives*. This is **L9** (gradients, not absolute values) seen from both sides

• upshot: Poisson **re-lights** the insert (seamless, but bleeds / erases flat regions); the pyramid **preserves** appearance and only dissolves the seam (robust, solver-free, but a true brightness mismatch lingers as a halo). Cloning → Poisson; panorama → pyramid

• [`fig-poisson-vs-laplacian-blend`: 1-D textured + constant source, and a 2-D constant square — Poisson vs pyramid]

#### Mixing gradients

• **mixing gradients** (Pérez 2003): per pixel, keep the **stronger** of the source's and destination's gradient, $\mathbf{v}=\arg\max(\lVert\nabla g\rVert,\lVert\nabla f^{*}\rVert)$, before solving — lets you insert **holey / lacy objects**, **transparent overlays**, or **text onto texture** so the destination's texture shows through the gaps. `fig-mixing-gradients`

• **interactive demo** (`fig-poisson-blend`): a seamless-cloning playground — upload/pick source+target, paint Ω, drag/scale the source, solve per-channel by SOR; switch color space (**linear / gamma / Chong log**) and toggle **mixed gradients**, with live source/dest/∇/reconstructed-∇/residual panels + a scanline. Standalone in `interactive-sandbox/poisson-blend/` (iframe-embedded, static PNG fallback), like the [[photo101-simulators]]. Ties together seamless cloning, the linear-vs-log encoding choice, and mixing gradients.

#### Illumination invariance, pixel encoding and color space

• **preferred solution = Chong's color space (Chong, Gortler & Zickler 2008)**: remind the reader of the transform from **RGB** (or **XYZ**) into Chong's space; because it is a **log** space it is **illumination-invariant** — the multiplicative shading becomes an **additive offset** that gradient-domain editing simply ignores, so equal gradients mean equal perceived contrast regardless of lighting. `fig-chong-colorspace-results` (results from their paper)

• **healing brush = a covariant derivative (Georgiev 2004), folded in**: Photoshop's healing brush is gradient-domain cloning made *multiplicative* — Georgiev replaces the plain gradient with a **covariant derivative** (work on $\log f$, transport gradients as ratios) so a patch matches the destination's *illumination*, not its absolute brightness — the same log/illumination-invariance point baked into the operator. Note the historical order: **Photoshop's healing brush came first (2002); Pérez et al. (2003) were inspired by it** and formalized the gradient-domain view.

#### Conservative-gradient simplification: the membrane shortcut

• **conservative-gradient simplification (Sylvain Paris / Farbman et al. 2009; Bhat 2010)**: when the guidance field is (or is made) **conservative**, the expensive Poisson solve collapses to adding a smooth **membrane** — solve $\nabla^2 u = 0$ (a Laplace, not Poisson, problem) for the boundary-mismatch correction $u$ and add it to the naive paste. Far cheaper, and the route to **real-time** gradient-domain editing.

#### Other gradient-domain edits

• **section framing — "cover your tracks"**: Poisson is the great tool for **hiding the evidence of an edit** — make any *local* edit blend so it leaves **no artifact at its boundary** (the destination flows smoothly out of the seam). Whatever you did inside the region, the gradient-domain reconstruction makes the join invisible.

• **panorama / seam blending**: stitch overlapping images by Poisson-blending across the chosen seam so exposure/color mismatches vanish at the join → forward-ref to the panorama / stitching part (where the seam is *found*; here it is *hidden*).

• **other gradient-domain edits**: HDR **tone compression** (attenuate large gradients, then integrate — Fattal 2002; integrate in log); **photomontage** (stitch many photos by choosing gradients + seams — Agarwala 2004); shadow / reflection removal; flash–no-flash fusion. All change only the **guidance field** $\mathbf{v}$ and reuse the same solver.

**Motivation — local tone mapping haloes.** `fig-tonemap-halo`

• the setup (craft callback): HDR **local tone mapping** compresses range by a **two-scale decomposition** — split into a large-scale **base** (the overall light) and a fine **detail** layer, compress the base, keep the detail, recombine (the *best-of-both* move, [[Style Reference]] rule 5)

• do it with a **plain Gaussian** base and it **bleeds across edges**: at a bright-sky / dark-mountain boundary the blur averages sky into the mountain, so after compressing the base a **bright halo** glows along the silhouette — the classic tone-mapping artefact

• *we want* a base layer that is smooth **within** a region but **stops at edges** — exactly an edge-preserving filter

• other motivations (state plainly, don't list-open): edits that must **not bleed past edges** — denoise without smudging detail; sharpen / abstract for stylization; build selection masks. Cross-link to [[#Compositing, segmentation and matting]] and stylization (NPR).

• **start from Gaussian blur** (name-then-use): the output is a weighted average of neighbours, weight = a spatial Gaussian $g_s(\lVert p-q\rVert)$ — close pixels count more. Illustrate on a **1-D scanline** = one row of the image. `fig-bilateral-1d`

• **sidebar — debugging the bilateral** (recipe moved here from Developing, testing & debugging): crank the **range variance $\sigma_r$ very large** → the range term drops out and the bilateral must collapse to a **plain Gaussian** (compare against your already-tested Gaussian); shrink **$\sigma_r$ very small** → pixels only average with near-identical neighbours, so **edges stay untouched** (run it on the half-plane / rectangle). Two extreme settings, two predictable outcomes that bracket the whole filter. `fig-bilateral-limits`

• **the problem**: near an edge the Gaussian **gathers information from across the boundary** — it averages the dark side into the bright side, smearing the step (the same bleed that made the halo)

• **the fix — a range weight**: multiply in a *second* Gaussian on the **value difference**, $g_r(\lvert I_p-I_q\rvert)$, that **down-weights pixels too different in value**. A neighbour that is close *in space* but far *in intensity* (across the edge) gets almost zero weight.

• **the bilateral weight** (Tomasi & Manduchi 1998): $w(p,q)=g_s(\lVert p-q\rVert)\,g_r(\lvert I_p-I_q\rvert)$, and the output is the normalized average $I^{\mathrm{bf}}_p=\frac1{W_p}\sum_q w(p,q)\,I_q$ with $W_p=\sum_q w(p,q)$. Read it back in words ([[Style Reference]] rule 3): *"average my neighbours, but only those that are both nearby and similar to me."* Two widths: $\sigma_s$ in pixels (how big a region), $\sigma_r$ in intensity units (how different is "too different").

• pseudocode block (from the slides): the brute-force double loop — precompute the **spatial** weight $g_s$ once, **recompute the range weight $g_r$ per pair** (it depends on the value difference, so it can't be tabulated or reused from convolution code), accumulate $\sum w\,I_q$ and $\sum w$, divide per pixel; truncate only the spatial Gaussian; RGB 3-D distance for color.

• **1-D demo**: a noisy **step edge** → Gaussian smears the step; the bilateral **denoises each plateau but keeps the step crisp**, because the weight collapses to one side at the edge. Then the **2-D image**: texture smoothed, edges held. `fig-bilateral-1d`, `fig-bilateral-2d`

• **the affinity reading** (the conceptual payoff): the range Gaussian *is* a similarity / **affinity** between two pixels — "how much do they belong together" — computed from their value difference. Every later method in this chapter is a variation on how to choose that affinity.

💡 **Big lesson:** edge-preserving = affinity — use the **color / intensity difference** between two pixels as how much they *belong together* (an **affinity**), then average (or connect) pixels in proportion to it. The bilateral's range weight $g_r$ is the first instance; the *same* affinity drives the bilateral grid, the guided filter, NL-means, colorization, matting and segmentation. *(L4 · first full edge-preserving treatment of the principle; registered in BASIC → Denoising, fully developed here.)*

#### Robustness for values with few neighbours

• **the failure**: a pixel whose value is **rare** (few neighbours within $\sigma_r$) has a tiny, noisy normalizer $W_p$ — the standard normalized bilateral becomes unstable / over-trusts the few similar pixels, and the Durand–Dorsey normalization is, strictly, **wrong** there

• **the fix — unnormalized / delta bilateral** (Aubry et al. 2014): express the output as a **delta from the input** rather than a fresh normalized average — $I^{\mathrm{bf}}_p=I_p+\sum_q w(p,q)\,(I_q-I_p)$ — and **drop the $1/W_p$ division**

• **why it works (intuition)**: each neighbour contributes a *correction* $w\,(I_q-I_p)$ pulling toward the input value; when few neighbours match, you make a **small correction** instead of dividing by a near-zero $W_p$ and amplifying noise — so a lonely value stays close to itself rather than blowing up

• ties to **local Laplacian filters** (Paris / Aubry): the unnormalized form is the bilateral cousin of the multiscale local Laplacian — see [[#Local Laplacian filters]] (and [[Linear pyramids and wavelets]])

#### Debugging — hand-checkable limits

• **large range variance** ($\sigma_r\to\infty$): $g_r\to1$, the range weight does nothing → the bilateral **collapses to a plain Gaussian** blur

• **tiny range variance** ($\sigma_r\to0$): only the pixel itself survives → output ≈ **identity**, edges (and everything) untouched

• these two limits are an **easy correctness test** for an implementation; a middle $\sigma_r$ interpolates between them

• also useful for tuning: pick $\sigma_s$ from the spatial scale of structure to remove, $\sigma_r$ from the contrast of edges to preserve (state the encoding first — see meta)

• **the twist**: read the **range weight from a *different* (guide) image** than the one being averaged — $g_r(\lvert J_p-J_q\rvert)$ with guide $J$, while still averaging the target $I$. The affinity comes from a *cleaner* signal.

• **why**: when the target is too noisy / dark for its own edges to be trusted, borrow edges from a **better-exposed guide**

• **flash / no-flash** (the headline app): smooth the noisy **no-flash** ambient image using **flash-image** edges as the guide — keeps the mood, kills the noise. Forward-ref → [[#Illumination related effects in a single image]] / flash–no-flash (Computational illumination). `fig-crossbilateral`

• a special case of the general **joint / guided** idea (→ guided filter, below)

• **the idea (Kopf et al. 2007)**: compute an **expensive** operation at **low resolution** — tone mapping, depth, stereo, colorization, a graph-cut label map — then **upsample it to full resolution guided by the full-res image**

• **how**: it's a cross/joint bilateral where the **range weight is read from the high-res guide** (the original image), so the cheap low-res solution snaps to the guide's edges and **stays sharp** — no blurry haloes across boundaries

• **why it matters**: a heavy solver runs on a tiny image, edges come (for free) from the big one — best-of-both, and the conceptual ancestor of HDRnet's learned edge-aware upsampling (below)

• **the idea** (Jiawen Chen's SIGGRAPH paper): turn the **nonlinear 2-D** bilateral into a **linear 3-D** filter by adding the **range / value as a third axis** — lift $(x,y)\to(x,y,I)$ into a coarse 3-D **grid**. `fig-bilateral-grid`

• **splat → blur → slice**: *splat* pixels into grid cells (accumulate value + weight); *blur* with a **plain separable 3-D Gaussian** (now an ordinary convolution — edges are automatic because dissimilar values live in different grid layers); *slice* back out by interpolating at each pixel's $(x,y,I)$

• pseudocode block (from the slides): **splat / blur / slice** as code — accumulate value and weight into cell $(x/\sigma_s,\,y/\sigma_s,\,I/\sigma_r)$; separable 3-D Gaussian on **both** grid and weight; slice by trilinear lookup at each pixel's $(x,y,I)$ and divide value by weight (that division *is* the bilateral normalizer $1/W_p$).

• **why it's fast**: the grid is **coarse** (downsampled in space and range), so it runs real-time on a GPU; ties to [[#Differentiable image pipelines and algorithm optimization (Halide)]] (Halide schedules exactly this pipeline)

• connects to the affinity lesson: the third axis *is* the affinity made geometric — pixels far apart in value are far apart in the grid

• **what the grid unlocks** (uses): (1) **cross / joint bilateral filtering** — splat the target, read the range axis from a guide; (2) **local histogram equalization** — each grid cell already stores a **local histogram** (the per-value counts splatted into it), so you can equalize / manipulate tone *locally* straight off the grid.

• **learn the grid**: a CNN predicts, at **low resolution**, the *coefficients* of a per-cell affine transform stored in a **bilateral grid**; slice + apply at **full resolution** → real-time learned enhancement on a phone

• the bilateral grid becomes a **learned, edge-aware upsampling** structure — best-of-both: a heavy network on a tiny image, edge-aware detail on the big one

• ties to L8 (a learned operator swaps a hand-designed prior for a learned one) — a one-line callback here, not a full box; xref ML / [[Machine learning]]

• **same affinity, made interactive**: the affinity that stops the bilateral from averaging across edges also powers an **edge-respecting selection / enhancement brush** — a manual local-edit brush whose painted effect **does not bleed across edges** (it follows the same value/color affinity)

• so it is the **interactive cousin of edge-preserving filtering**: where the bilateral decides automatically "these pixels belong together," the brush lets a user *paint* an edit that respects exactly the same boundaries — dodge/burn, local tone or color tweaks that stop at the object's edge.

• the one idea: a cut is **least noticeable where the image is already changing** (a strong edge or busy texture) and **most noticeable in smooth regions** — so *seek* edges to hide the seam, the mirror image of edge-preserving smoothing.

• the **panorama motivation** (craft callback): naive reprojection shows a hard seam, smooth/feather blending **ghosts** moving objects; the fix is to **route the seam around disagreement** — build a difference map of "bad things happen here," then find the boundary that crosses the fewest changing pixels. (Pays off the panorama-blending forward-ref to graph cut.)

• **two regimes**: **interactive** path-finding where a user guides the cut (scissors, snakes), vs **automatic** global optimization (seam carving, graph cut, normalized cuts). Present the automatic ones as the main line; scissors/snakes as the interactive ancestors.

• **the idea** (Mortensen & Barrett 1995, *name-then-use*): an interactive selection tool — the user clicks a seed on an object boundary; as the cursor moves, a **"live wire" snaps to the strongest nearby edge** and traces the boundary in real time. Click to anchor, continue.

• **mechanism = shortest path on a cost graph**: pixels are nodes; edge cost is **low where image gradient is high** (cost $\propto$ inverse edge magnitude, + gradient-direction terms). Dijkstra from the seed gives the **least-cost path** to the cursor — the DP/shortest-path idea made interactive. `fig-snake-livewire`

• the payoff: same machinery as seam carving (least-cost path), different goal — **select/cut along** an edge rather than carve through a non-edge. The cost function is the design.

• **the idea** (Kass, Witkin & Terzopoulos 1988): a deformable **curve** that **relaxes onto an object boundary** by minimizing an energy — the variational, continuous counterpart to live-wire's discrete path.

• **the energy** (read it back): $E=\int E_\text{int}+E_\text{image}+E_\text{con}$ — **internal** keeps the curve smooth (elastic tension $\alpha$, bending stiffness $\beta$); **image** $E_\text{image}=-\lvert\nabla I\rvert^2$ pulls it toward **strong edges**; **constraint** lets the user nudge it. Minimized by gradient descent / a sparse banded solve — same linear-algebra machinery as the regression chapter.

• honest tradeoff: snakes are **local** (need a good initialization, can get stuck) — which motivates the **global** optimizers next; note balloon/GVF/level-set variants only in passing.

• **the leap from local to global** (Boykov & Jolly 2001): pose the cut as a **binary labeling** — every pixel is **object** or **background** — and minimize a **graph energy** whose minimum is found **exactly** (binary case) by **min-cut = max-flow**. No local minima, unlike snakes.

• **the energy** $E(L)=\sum_p D_p(L_p)+\sum_{(p,q)}V_{pq}(L_p,L_q)$ (read back): a **data term** $D_p$ — how well pixel $p$ fits each label (from user scribbles / color models) — plus a **smoothness term** $V_{pq}$ — a penalty for neighbours getting *different* labels, made **small across true edges** so the boundary **prefers to cut along edges**. (This $V_{pq}$ **is** the L4 affinity / the bilateral range weight — callback.) `fig-graphcut-segmentation`

• **the graph construction**: pixels = nodes; two terminals **source (object) / sink (background)**; **t-links** carry $D_p$, **n-links** carry $V_{pq}$; the **min cut** severs the cheapest set of edges separating source from sink → the segmentation. Solve by **max-flow** (Boykov–Kolmogorov 2004).

• **multi-label, in passing**: more than two labels (photomontage, stereo, denoising) is NP-hard exactly — solved approximately by **$\alpha$-expansion** (Boykov, Veksler & Zabih 2001), iterated binary graph cuts. Note, don't develop.

• **GrabCut** (Rother, Kolmogorov & Blake 2004): graph-cut segmentation made **easy to drive** — the user drags **one rectangle**, iterated graph cuts + updated **GMM color models** extract the object. (Cross-ref the segmentation/matting section, which owns the *application*.)

• **the problem with plain min-cut**: minimizing the raw cut weight is **biased toward tiny cuts** — it happily lops off a single isolated pixel. We want **balanced** partitions.

• **the fix** (Shi & Malik 2000): normalize the cut by each side's total **association** — $\mathrm{Ncut}(A,B)=\frac{\mathrm{cut}}{\mathrm{assoc}(A,V)}+\frac{\mathrm{cut}}{\mathrm{assoc}(B,V)}$ — penalizing unbalanced splits. The affinity $w_{uv}$ is again the **L4 affinity** (color / proximity similarity).

• **how it's solved (intuition only — keep eigen-machinery light, V13)**: exact Ncut is NP-hard, but it **relaxes to a generalized eigenproblem** $(D-W)x=\lambda D x$; the **second-smallest eigenvector** gives a soft partition you threshold — a **spectral** (vs combinatorial) route to the cut. Defer eigen-fluency to the appendix.

• the contrast worth stating: graph-cut = **combinatorial, needs seeds, exact binary**; normalized cut = **spectral, unsupervised, approximate** — two answers to "where's the best boundary," same affinity input.

• **the application** (Avidan & Shamir 2007, motivation-first): resize a photo to a new aspect ratio **without squashing the subject or cropping it off** — instead of uniform scaling or cropping, **remove (or insert) the least-important pixels**, one connected seam at a time. `fig-seam-carving`

• **a seam** = an 8-connected top-to-bottom (or left-to-right) **path of one pixel per row** whose total **energy** $\sum e(i,j)$ is **minimum**; energy $e=\lvert\partial_x I\rvert+\lvert\partial_y I\rvert$ (gradient magnitude / saliency) — so the seam **threads through low-energy, low-detail regions** and avoids the subject.

• **the DP** (the worked mechanism, `fig-seam-energy-dp`): fill a **cumulative-energy** table top-down, $M(i,j)=e(i,j)+\min\!\big(M(i{-}1,j{-}1),M(i{-}1,j),M(i{-}1,j{+}1)\big)$; the cheapest endpoint $\arg\min_j M(H,j)$ + **back-trace** = the optimal seam. Remove it (narrow), or duplicate it (widen); **repeat**. Read back: *"the cheapest way down is the cheapest way to my best parent, plus my own cost."*

• pseudocode block: the DP sweep that fills $M$ row-by-row with the 3-way $\min$, then the $\arg\min$ + upward back-trace that reads off the seam.

• **the same DP as live-wire/scissors** (through-line): a 1-D least-cost path on a gradient-derived energy — only the **energy** and the **direction of the cut** differ.

• honest limits: seam carving **fails on busy/structured images** (no low-energy path exists → it carves through content); better saliency/forward-energy helps — note, don't develop.

• **the idea**: choosing frames/exposures for a smooth **time-lapse or day-to-night sequence** is the **same least-cost path**, but the axis is **time** — pick a sequence minimizing inter-frame jumps (exposure/color discontinuity), a DP over candidate frames. ⚠️ *coverage note: no dedicated slide/transcript found — confirm scope or trim to a one-line example under seam carving.*

• **the idea** (Schödl et al. 2000): turn a short clip of a stochastic phenomenon (waving flag, candle, fountain) into an **arbitrarily long, seamless loop** by finding **good transitions** — frames that look similar enough to jump between invisibly.

• **mechanism = a cut in the time graph**: frames are nodes; a transition cost = **inter-frame similarity** (low cost = invisible jump); finding loops / a play order is **least-cost path / DP** on this graph — the seam idea applied to **time**. `fig-video-texture-loop`

• the payoff: same "hide the seam where things match" principle as panorama blending and seam carving — now between **frames** instead of pixels.

• **GraphCut textures** (Kwatra et al. 2003): synthesize/extend a texture (or composite two images) by **graph-cut along the best seam** through the overlap region — the min cut picks the path where the two sources **agree best**, so the join is invisible. The texture-synthesis sibling of panorama seam-finding.

• **interactive digital photomontage** (Agarwala et al. 2004 — **shared with Poisson**): the headline **"cut + blend" pipeline** — **graph cut** chooses *which source contributes each pixel* (the seam, this section), then a **gradient-domain (Poisson) blend** hides any residual seam (the reconstruct-from-edges section). The clean statement of how the three legs **compose**: cut along the edges, then reconstruct across the seam. (Explicit two-way xref to [[#Poisson image editing]].)

• **MRFs**: the graph-cut energy $E(L)=\sum D_p+\sum V_{pq}$ **is** a pairwise **Markov Random Field** — data + pairwise prior — and graph cut / belief propagation are its inference engines. Name the connection (so the reader meets "MRF" once); the broader Bayesian/MRF treatment lives with inverse problems / segmentation, cross-ref rather than develop here. *(Resolves the old "MRFs? / Bill's Bayesian stuff" stub.)*

• closing through-line: scissors · snakes · seam carving · graph cut · normalized cuts · video textures are **one idea wearing different optimizers** — *cut where it won't be seen; the energy is the design.*

• **reconstruct *from* edges — Poisson / gradient-domain**: throw away absolute level, keep gradients, integrate back. Reach for it to **hide an edit's boundary** — seamless cloning, healing, gradient-domain HDR, photomontage/panorama blending.

• **smooth *except across* edges — bilateral & friends**: average pixels that *belong together* (the L4 affinity). Reach for it to **denoise / decompose / enhance without haloes or bleeding** — bilateral, joint/cross & upsampling, grid, guided filter, local Laplacian, NL-means, LARK, colorization.

• **cut *along* edges — seam optimization**: find the least-noticeable path/boundary. Reach for it to **resize, select, or composite** by hiding a seam in the busy regions — seam carving, scissors/snakes, graph cut, normalized cuts, GraphCut textures.

• **Poisson**: global, seamless, encoding-sensitive; cost = a **sparse linear solve** (CG / multigrid / FFT; membrane shortcut when conservative). Pick when the artifact lives **at a boundary**.

• **bilateral & friends**: mostly a **single pass** (grid → real-time; guided $O(N)$; local Laplacian heavier; colorization a solve). Pick when you must **process within regions but not across edges**. Watch range-quantization / gradient-reversal artefacts (guided & local Laplacian avoid them).

• **seam optimization**: **discrete optimization** — DP (1-D path, cheap), graph cut (2-D, exact binary, needs seeds), normalized cut (spectral, unsupervised, approximate). Pick when the right answer is **where to put the boundary**; fails on busy images with no low-energy path.

• **they compose**: photomontage / panorama = **graph-cut seam + Poisson blend** (cut along edges, then reconstruct across the seam) — the part's payoff.

• **the one-line definition**: a warp applies a function $f:\mathbb{R}^2\to\mathbb{R}^2$ to the image *domain* — "if $(x,y)$ had color $c$ in the input, then $f(x,y)$ has color $c$ in the output." It deforms **where pixels live**, leaving their **colors (the range) untouched**. Contrast a *point/tone operation*, which edits the range (colors) and leaves the domain fixed. Warping is the domain twin of everything in BASIC point-ops. [`fig-warp-domain-vs-range`]

• **the rubber-sheet picture**: imagine the image printed on rubber and stretched — pixels slide to new positions; the picture content rides along. This is the right mental model for *all* of it, from a tiny lens-distortion correction to "liquify"/slimming, to morphing, to a full perspective rectification.

• **why it's everywhere** (motivate before machinery): geometric tasks across this whole part are warps — **optical aberration / lens-distortion correction**, **video stabilization**, **panorama reprojection** (fix parallax / align to a reference), **wide-angle distortion correction**, **morphable face models**, **image-based rendering**, and the cosmetic "slim the subject." One primitive, reused everywhere — which is exactly why this chapter comes first in the part.

• **two-and-a-half hard questions** (the chapter's skeleton, stated up front): given the rubber-sheet idea, three things are genuinely non-trivial — (1) *which direction* do we transform (forward or inverse)? (2) *how do we read a color at a non-integer location* (resampling/reconstruction)? and (2.5, only for free-form warps) *how do we even specify $f$* from a handful of user inputs? The next three sections take these in turn.

• **forward warp (input-driven)**: loop over every *input* pixel $(x,y)$, compute its destination $f(x,y)$, and write its color there: $\text{out}(f(x,y))\leftarrow\text{in}(x,y)$. Intuitive — "push each pixel to where it goes." But it has **two failure modes** baked in. [`fig-forward-warp-holes`]

• **holes / gaps**: destinations $f(x,y)$ are generally **non-integer** and **non-surjective** — when the warp stretches a region, neighbouring inputs land far apart, leaving output pixels that **no input ever writes** (black speckle / cracks).

• **pile-ups / double-writes**: where the warp compresses a region, **many** inputs land on the **same** output pixel — who wins? You'd have to splat-and-accumulate with weights and normalize, which is fiddly and still doesn't fill the gaps.

• **inverse warp (output-driven)** — *the useful one*: loop over every *output* pixel $(x,y)$, push it **backward** through $f^{-1}$ to find where it came from in the input, and **look up** that color: $\text{out}(x,y)=\text{in}\big(f^{-1}(x,y)\big)$. [`fig-inverse-warp-lookup`]

• **why it wins**: the main loop is over **output** pixels, so **every output is covered exactly once** — no gaps, no double-writes by construction. The non-integer problem is pushed entirely into the *input* lookup, where it's a clean **resampling** question (next section) rather than a scatter problem.

• **take-home (state it as a rule)**: *main loop over OUTPUT pixels; use the INVERSE transform; resample the input.* This is the canonical resampling/warping idiom and recurs verbatim in panorama warping ([[Manual panorama stitching from multiple views]]) and rectification.

• **pseudocode**: a three-line block (loop output pixels → `(x',y')=f⁻¹(x,y)` → `out[x,y]=resample(in,x',y')`) makes the output-driven loop and once-per-pixel coverage concrete.

• **the catch — you need $f^{-1}$**: inverse warping needs the *backward* map. For **parametric** warps this is free (invert a matrix; for a homography, use $H^{-1}$). For a warp you only know **forwards** — most importantly a **dense optical-flow field** (each input pixel's motion) — you don't have $f^{-1}$ in closed form, and either invert the field numerically or fall back to forward splatting. So forward warping isn't *wrong*, it's the tool when the field is only available input-side. (Cross-ref [[Optical flow]] — rendering a frame along an estimated flow is exactly this case.)

• **boundary handling**: $f^{-1}(x,y)$ can land **outside** the input. Decide the policy — **black/zero padding**, **clamp-to-edge** (replicate the border pixel), **mirror**, or **mark as undefined** (transparent / alpha). Rectification and stabilization both produce outputs that read past the input border (the corners), so this is not a corner case — it's the common case.

• **the core problem**: inverse warping asks for $\text{in}$ at a **fractional** coordinate $f^{-1}(x,y)$. An image is *samples* of an underlying continuous signal; to read between samples we **reconstruct** the continuous signal and evaluate it — this is *resampling* (reconstruct, then re-sample). The reconstruction filter is the whole story.

• **nearest neighbour**: round to the closest sample. Cheapest; **blocky / jaggy** (a box reconstruction kernel) — visible stair-steps under rotation/scaling. Fine only for label maps / when you must not invent in-between values. [`fig-recon-filters-1d`, "nearest"]

• **bilinear**: linear interpolation in 1-D, $\text{im}(x)=(1-\alpha)\,\text{im}[\lfloor x\rfloor]+\alpha\,\text{im}[\lceil x\rceil]$ with $\alpha$ the fractional part; in 2-D it's **separable** — interpolate along $x$ on the two top and two bottom neighbours, then along $y$ (or $y$ then $x$ — same result, a worthwhile exercise). Equivalently the four-neighbour **area weights**. Smooth, cheap, the workhorse default; slightly soft (a triangle/tent kernel attenuates high frequencies). [`fig-bilinear-weights`]

• **bicubic / Lanczos** — *the right framing*: do **not** think "fit a cubic to the data." Think **convolution with an analytic kernel**: $\text{out}(x)=\sum_{x'}k\big(f^{-1}(x)-x'\big)\,\text{in}(x')$, the kernel $k$ centred at the (fractional) source location and sampled at the surrounding integer inputs. **Bicubic** (the Mitchell–Netravali family) uses a wider, sharper, negative-lobed kernel over a $4\times4$ neighbourhood — sharper than bilinear, with slight ringing/overshoot you can tune. **Lanczos** is a windowed sinc, sharpest of the common kernels (more ringing). All are **separable** (run in $x$, then $y$), and in general the weights **can't be precomputed** because the fractional offset differs per output pixel (though for integer or rational scale factors they become periodic and *can* be tabulated). [`fig-recon-filters-1d`]

• **upsampling vs downsampling is the key split** (lead-in to L16): the kernel-as-convolution view makes the difference precise. **Upsampling (enlarging)**: reconstruct and read — clamp the kernel width to 1 input pixel; *no* prefilter needed (you're not discarding anything). **Downsampling (minifying)**: you're **discarding** samples, so reconstruction alone **aliases**.

• 💡 **the L16 discipline** — *prefilter before you downsample* (see the boxed lesson above): blur/area-average **before** decimating. In the convolution-resampling view this is one clean move: **scale the kernel up by the downscale factor**, $k_s(t)=\tfrac1s\,k(t/s)$, and renormalize — now each output pixel **gathers over the whole footprint** it covers in the input, i.e. averages instead of point-sampling. The infamous bug is **keep-every-Nth decimation** (or a width-1 kernel on minify): fine detail folds into bold **moiré / herringbone**. The honest rule: *upsampling clamps the kernel to width 1; downsampling widens it proportionally to scale.* [`fig-minify-aliasing-moire`, `fig-kernel-scaling-minify`]

• **practical notes**: resampling is **separable** — scale/resample in $x$ (producing an anisotropically-sized intermediate image), then in $y$; downsampling-in-$y$-first vectorizes well. Implementation-wise the clean trick (Adams) is to **form the tiny sparse 1-D resampling matrix** (one row/column's worth of weights, with boundary handling and renormalization folded in) and apply it to every row/column — the weight matrix is negligible next to the image. **For minification, MIP-maps** (a prefiltered pyramid) precompute the area-averages once and read them back per query — the standard real-time answer to L16. (Cross-ref [[Linearity, Fourier, Aliasing and deblurring]] for the Nyquist/aliasing account; pyramids for the band structure.)

• **the honest limit of interpolation** (forward-ref): resampling adds *pixels*, never new *frequencies* — bicubic/Lanczos upsampling looks soft because the high frequencies were never measured. Recovering plausible detail is **super-resolution**, a prior-driven inverse problem, not resampling (cross-ref [[Super-resolution and image priors]] — reconstruction vs hallucination).

• **the idea**: for many warps $f$ is a **low-parameter** function — fit it to a few correspondences and apply it everywhere. Climbing the **degrees-of-freedom ladder**, each rung adds freedom and **breaks a preserved property**: [`fig-dof-ladder`]

• **translation** (2 DOF) — preserves everything but position.

• **rigid / Euclidean** (3) — + rotation; preserves lengths & angles.

• **similarity** (4) — + uniform scale; preserves angles & ratios (shape).

• **affine** (6) — $\binom{x'}{y'}=A\binom{x}{y}+\mathbf t$; preserves **parallelism** and **ratios along lines**, *not* angles/lengths. Straight lines stay straight; a square → a parallelogram.

• **projective / homography** (8) — $\binom{x'}{y'}{w'}=H\binom{x}{y}{1}$ then divide by $w'$; preserves **straight lines only** — parallels may **converge** (a square → an arbitrary quadrilateral). This is the rung that does **perspective**, and the subject of the next chapter and of panorama stitching.

• **why homogeneous coordinates** (link, don't re-derive): the homography is non-linear in $(x,y)$ because of the **division by $w'$**, but **linear** in homogeneous coordinates — a single $3\times3$ matrix multiply followed by the perspective divide. It *is* the reprojection of a plane under a 3-D rotation (project, rotate, reproject), which is why "depth doesn't matter for a planar-or-rotation warp." Full development: FUNDAMENTALS — Pinhole; the **solve** for $H$ from correspondences: [[Manual panorama stitching from multiple views]].

• **fitting a parametric warp**: each point correspondence gives **two equations**; an affine needs **3** correspondences (6 unknowns), a homography **4** (8 unknowns up to scale). More correspondences → **overdetermined** → **least squares** (or SVD for the homography null-vector). This is a clean instance of [[Linear Inverse Problems and Regression]] — *the warp parameters are the regression unknowns.* (Robustness to wrong matches → RANSAC, developed for stitching.)

• **strength and limit**: parametric warps are **global, smooth, few-parameter, robust** — perfect when the deformation really is rigid/affine/projective (a plane, a camera rotation, a lens model). They **cannot** express a local, free-form deformation — a smile, a morph, a hand-drawn stretch. For that, you need a field built from sparse control (next).

• **the setup**: the user gives a **sparse** set of correspondences — a few control points, or pairs of feature segments — and we must produce a **dense** warp field $f$ defined at every pixel by *interpolating* the sparse constraints. Three standard constructions, trading smoothness, locality, and control:

• **mesh / triangulation warp (piecewise-affine)**: triangulate the control points (e.g. Delaunay); inside each triangle the three vertex correspondences **exactly determine an affine map** (6 DOF, 3 points), so warp each triangle by its own affine. Simple, fast, exact at the controls, and the standard morphing workhorse. Downside: **$C^0$ only** — the field is continuous but **creases** at triangle edges (derivative jumps), and quality depends on the triangulation. [`fig-mesh-triangulation-warp`]

• **RBF / thin-plate-spline (TPS) warp (smooth scattered-data interpolation)**: fit a **smooth** field that interpolates the control displacements with **minimal bending energy**: $f(\mathbf p)=A\mathbf p+\mathbf t+\sum_i w_i\,\phi(\lVert\mathbf p-\mathbf c_i\rVert)$, an affine "global" part plus a sum of **radial basis functions** centred at the controls. The thin-plate kernel $\phi(r)=r^2\log r$ minimizes integrated second-derivative energy — the smoothest interpolant (the physical analogy: a thin metal plate bent to pass through the constraints). The weights $w_i$ (and $A,\mathbf t$) solve a small **linear system**. Smooth everywhere ($C^1$+), no creases; **global support** (every control influences every pixel, so it can be less local than a mesh) and an $O(n^2)$–$O(n^3)$ solve in the number of controls. The go-to for smooth medical/face registration and morphometrics (Bookstein). [`fig-tps-rbf-warp`]

• **feature-based field warping — Beier & Neely 1992**: specify the warp with **pairs of directed line segments** (before/after) rather than points — natural for features like an eyebrow or a jawline. Each segment pair $(\mathbf{PQ}\to\mathbf{P'Q'})$ defines a **local coordinate frame**: a coordinate $u$ **along** the segment (normalized, $0$ at $\mathbf P$, $1$ at $\mathbf Q$) and $v$ **perpendicular** (in distance units). For a single segment, map a point by reading its $(u,v)$ in the **destination** segment's frame (inverse warp) and reconstructing it in the source frame:

• $u=\dfrac{(\mathbf X-\mathbf P)\cdot(\mathbf Q-\mathbf P)}{\lVert\mathbf Q-\mathbf P\rVert^2}$, $\quad v=\dfrac{(\mathbf X-\mathbf P)\cdot\perp(\mathbf Q-\mathbf P)}{\lVert\mathbf Q-\mathbf P\rVert}$; then $\mathbf X'=\mathbf P'+u\,(\mathbf Q'-\mathbf P')+\dfrac{v\,\perp(\mathbf Q'-\mathbf P')}{\lVert\mathbf Q'-\mathbf P'\rVert}$.

• **note** (Beier–Neely's own finding): $u$ scales with the segment (stretch), but $v$ is kept **absolute** — they tried scaling $v$ too and it looked worse.

• **multiple segments**: each segment $i$ proposes a displaced point $\mathbf X'_i$; the result is a **distance-weighted average**, weight $w_i=\big(\ell_i^{\,p}/(a+d_i)\big)^{b}$ — longer segments ($\ell_i$) carry more influence, nearer ones ($d_i$ = distance to the segment) dominate; $a,b,p$ tune the falloff. [`fig-beier-neely-segments`]

• **provenance / use**: the technique behind Michael Jackson's *Black or White* morph; a classic course problem set. *Built here, used in [[Morphing]].* (Ref: Beier & Neely 1992; the canonical warping text is Wolberg 1990.)

• **as-rigid-as-possible (ARAP) / MLS warps — penalize shear**: the catch with the above is that interpolating positions alone lets shapes **shear and squash** unnaturally when you drag a handle. **As-rigid-as-possible** (Igarashi 2005) and **moving-least-squares** rigid/similarity deformation (Schaefer 2006) add an explicit objective: each local neighbourhood should transform **as close to a rotation+translation (rigid) as possible** — minimize the deviation from rigidity rather than just matching control points. The deformation then **preserves local shape** (a dragged limb bends but doesn't stretch), which is what makes interactive shape manipulation and detail-preserving warps look natural. Conceptually: same "dense field from sparse handles," but with a **regularizer on local rigidity** instead of plain smoothness. [`fig-arap-vs-affine`]

• **choosing** (decision aid): a **plane / camera rotation / lens model** → parametric (affine/homography), fit by least squares. A **face/shape morph** with creases acceptable and speed wanted → **mesh/triangulation**. A **smooth** registration with few controls → **TPS/RBF**. **Feature-line** control (eyebrows, jaw) → **Beier–Neely**. **Interactive handle-dragging** where shapes must not shear → **ARAP/MLS**. Whatever the choice, the *application* is identical: produce $f$, then **inverse-warp and resample** (the first three sections) — that primitive never changes.

• **the goal**: *smoothly* turn one image into another — a face into another face, one video keyframe into the next (slow-motion / frame interpolation). "Smooth" means the in-between $I_t$ should look like a *plausible single image*, not a blend of two.

• **the naive attempt — cross-fading (a.k.a. cross-dissolve)**: linearly interpolate colors pixel-by-pixel, $\;\text{out}[y,x] = (1-t)\,I_0[y,x] + t\,I_1[y,x]$. This is the right idea for **color** but ignores **position**.

• **why it ghosts**: the two images' **features are not aligned** — eyes, mouth, silhouette sit at different pixels. Blending colors at fixed locations super-imposes a left-image eye on a right-image cheek → a translucent **double exposure** (four eyes, two mouths). And you generally *cannot* fix this with one global alignment (translate/rotate/scale) — faces differ locally, non-rigidly. [`fig-morph-crossfade-ghost`]

• **the diagnosis**: cross-fading interpolates only the **range** (color). To get a believable in-between we must *also* interpolate the **domain** — the **location** of every feature. This is a **domain transform** (a warp), and it's the missing half.

• 💡 **the throughline**: *separate shape and color, and interpolate each.* Everything below is how to (a) get a dense **shape** interpolation from sparse correspondence and (b) combine it with the **color** interpolation. (See the *align before you blend* big-lesson box above.)

• **range interpolation (color)** — the cross-dissolve we already have: $\;C(x) = (1-t)\,C_0(x) + t\,C_1(x)$. A straight linear blend of pixel values.

• **domain interpolation (location)** — treat a feature **point** as a vector and interpolate *where it is*: for corresponding points $P$ (in $I_0$) and $Q$ (in $I_1$), the in-between location is $\;V = (1-t)\,P + t\,Q$. At $t=0$ the feature is where $I_0$ put it; at $t=1$ where $I_1$ put it; in between it travels along the straight segment. [`fig-morph-shape-vs-color`]

• **warping = a domain transform**: moving pixels spatially while leaving their colors alone, $\;C'(x,y) = C\big(f(x,y)\big)$. In practice we use the **inverse / backward** map — for each output pixel, look up where it came from: $\;\text{out}(x,y) = \text{im}\big(f^{-1}(x,y)\big)$ — so every output pixel is filled (no holes) and resampling is well-defined. This is exactly the engine of [[Warping and resampling]]; morphing is one of its biggest customers (it warps twice per frame).

• 💡 **the morph in one line**: *interpolate the domain like a vector, then interpolate the range (color).* A morph is the **product** of a shape interpolation and a color interpolation — two independent linear blends applied together. [`fig-morph-shape-vs-color`]

• **inputs**: two images $I_0$ and $I_1$, plus **sparse correspondences** the user specifies on both (matching line segments, or matching mesh vertices — *which* primitive is the next two sections). Output: a sequence of in-betweens $I_t$, $t\in(0,1)$.

• **per intermediate frame $I_t$** (the canonical four steps):

1. **interpolate the feature geometry** to an intermediate shape: each corresponding pair (point/segment/vertex) $\to$ its in-between location $P_t = (1-t)\,P_0 + t\,P_1$.

2. **build a dense warp field** from the sparse pairs — two of them: one taking $I_0$'s features onto the intermediate shape, one taking $I_1$'s features onto the *same* intermediate shape.

3. **warp both images** to that common shape. Now $I_0$ and $I_1$ are **feature-aligned**: both pairs of eyes land on the same pixels, both mouths coincide.

4. **cross-dissolve** the two warped images: $I_t = (1-t)\,\text{warp}(I_0) + t\,\text{warp}(I_1)$. Because they're aligned, the blend is sharp — **no ghosting**. [`fig-morph-recipe`]

• **why warp *both* (not just one)**: warping only $I_0$ all the way to $I_1$'s shape and dissolving would make features *slide* while colors fade — and would over-distort early/late frames. Meeting **in the middle** (both warp partway to the shared intermediate shape) keeps distortions small and symmetric, and is what makes the motion read as one object transforming.

• **endpoints check**: at $t=0$ the intermediate shape = $I_0$'s shape, warp of $I_0$ is identity, color weight on $I_1$ is 0 → output is exactly $I_0$ (and symmetrically at $t=1$). The morph is a *genuine* interpolation, not just a fancy blend.

• **note (cross-ref)**: the correspondences here are **user-specified and sparse**. The **automatic, dense** way to get correspondence between two images is [[Optical flow]] — that's exactly what frame-interpolation / slow-motion methods use to morph between *video* frames without hand-drawn lines ([[Frame interpolation and slow-motion synthesis]]).

• **the primitive**: a **pair of directed line segments** — one drawn on the "before" image, one on the "after" — saying *"this feature edge here corresponds to this feature edge there."* A handful of segments (jaw, eyes, nose bridge, hairline) is enough; the warp interpolates everywhere else. (Beier & Neely 1992 — "Feature-Based Metamorphosis"; the technique behind Michael Jackson's *Black or White* video and a classic pset.)

• **one segment defines a coordinate change**: a before/after pair $\overline{PQ}\to\overline{P'Q'}$ induces a similarity (rotate + scale + translate) — the simplest transform that carries one segment onto the other. Set up a **segment-relative coordinate system**: along the segment, a **normalized** coordinate $u\in[0,1]$ (0 at $P$, 1 at $Q$); perpendicular to it, a **signed distance** $v$ (in pixels). For a query point $X$:

$$u = \frac{(X-P)\cdot(Q-P)}{\lVert Q-P\rVert^2}, \qquad v = \frac{(X-P)\cdot \text{perp}(Q-P)}{\lVert Q-P\rVert},$$

where $\text{perp}(\cdot)$ is the 90°-rotated vector. Then the corresponding point in the *other* image is reconstructed from the *primed* segment:

$$X' = P' + u\,(Q'-P') + \frac{v\cdot\text{perp}(Q'-P')}{\lVert Q'-P'\rVert}.$$

Read it: **$u$ scales** with the segment (a feature twice as long stretches with it), but **$v$ stays in absolute pixel units** (perpendicular offset is *not* rescaled — Beier & Neely report that scaling $v$ too looked worse). [`fig-beier-neely-oneseg`]

• **inverse-map direction**: you compute $(u,v)$ in the **destination** ("after") frame and look up the source — because warping uses the **inverse** map ($\text{out}(x)=\text{im}(f^{-1}(x))$), so each output pixel is filled.

• **multiple segments → a weighted average of displacements**: each line pair $i$ proposes its own target $X'_i$. Combine them by a **distance-weighted average**, where a line's influence falls off with the query point's distance to that segment and grows with the segment's length:

$$X' = \frac{\sum_i w_i\,X'_i}{\sum_i w_i}, \qquad w_i = \left(\frac{\text{length}_i^{\,p}}{a + \text{dist}_i}\right)^{b}.$$

The tunables $a,b,p$ control smoothness ($a$: how strongly far lines still pull), falloff ($b$), and the length bias ($p$). Distance-to-a-segment is the usual capped point-to-segment distance (perpendicular within the span, endpoint distance outside). [`fig-beier-neely-multiseg`]

• **strengths / weaknesses**: *strength* — sparse, intuitive control (draw a few feature lines), smooth fields, no mesh to manage. *weakness* — every line affects every pixel (global support → $O(\#\text{pixels}\times\#\text{lines})$ per frame), and far-apart lines can fight, giving "ghost" pulls and folds; careful $a,b,p$ tuning and debugging the distance function are part of the work.

• **bells & whistles**: morph the **foreground only** (matte it first) so the background doesn't drag artifacts; **non-uniform** timing (different features morph at different rates) for more lifelike transitions.

• **the primitive**: instead of lines, place a **mesh of corresponding control points** on both images (or a regular grid deformed to match features). Triangulate the points; each triangle in $I_0$ maps to the matching triangle in $I_1$.

• **the warp**: within each triangle the map is a single **affine** (barycentric) transform — fast, local, and exactly invertible. The intermediate shape is the mesh with vertices at $P_t=(1-t)P_0+tP_1$; warp each source by its per-triangle affine onto that mesh, then dissolve. (Spline/free-form mesh variants — Wolberg 1990; multilevel free-form, Lee et al. 1996 — trade the piecewise-affine creases for smoother fields.)

• **field vs mesh — the trade**: *mesh* warps have **local support** (a vertex moves only its incident triangles → fast, predictable, no far-line fights) but the control net is fiddly and triangle edges can crease; *field (Beier–Neely)* warps have **global, line-based** control (sparse and intuitive) but are slower and can fold. Same job — *turn sparse correspondence into a dense, invertible warp* — two different bases. [`fig-mesh-vs-field-morph`]

• **either way, the same resampling**: each warp is a domain transform that must **reconstruct + prefilter** as it resamples (minified regions averaged, magnified regions smoothly interpolated) — the EWA/mip machinery of [[Warping and resampling]]. Morphing is a stress test for good resampling.

• **when a 2-D morph is *wrong***: if $I_0$ and $I_1$ are two **photographs of the same 3-D scene** from different viewpoints (e.g. a face turning), a naive linear morph **bends straight lines and shrinks the object** — the in-between is not a valid view of *any* rigid scene. Linearly interpolating image positions is **not** the same as interpolating the underlying 3-D geometry. [`fig-view-morph-prewarp`]

• **the fix (Seitz & Dyer 1996, *View Morphing*)** — three steps that make the morph a *physically valid* intermediate view:

1. **prewarp** both images by homographies that **rectify** them to a common plane (so the two image planes are parallel and corresponding **epipolar lines** are horizontal and aligned);

2. **linearly interpolate** the now-rectified images (a plain morph — but in this rectified frame the linear interpolation *is* geometrically correct, because corresponding points move along aligned scanlines);

3. **postwarp** the result by an interpolated homography to the desired in-between view.

• **why it works**: rectification turns the general two-view geometry into the **parallel-camera (pure-translation) case**, where image-space linear interpolation exactly corresponds to moving the camera along the baseline — straight lines stay straight, the object keeps its shape, and the in-between is a true perspective view. It's the projectively-aware upgrade of the plain morph.

• **the connection to flow / stereo**: view morphing needs **corresponding points** between the views and the epipolar geometry relating them — exactly the **correspondence** problem of [[Optical flow]] and stereo, and the **epipolar** constraint of [[Image alignment]] / multi-view. With dense correspondence you can synthesize in-between *views* automatically — the bridge from morphing to **image-based rendering** and to flow-driven **frame interpolation** ([[Frame interpolation and slow-motion synthesis]]).

• **the lasting ideas**: (1) **linear interpolation of color alone introduces blur/ghosts** — you must interpolate position too; (2) **separate shape and color**, then recombine; (3) **non-rigid alignment** of different images is a reusable primitive. [`fig-morph-recipe`]

• **where it goes**: **special effects** (the canonical face-morph); **video frame interpolation / slow-motion** and even **MPEG-style** motion prediction (morph-by-motion-field); **morphable models** for face recognition/animation (warp a mean face by shape parameters); **motion magnification** (Liu et al. 2005 — *analyze* a motion field, **magnify** it, and warp by the magnified field — a morph whose correspondence comes from motion analysis, robust to occlusion). The common thread: once you can **warp one image into correspondence with another**, you can interpolate, exaggerate, average, or re-render — and *getting* that correspondence automatically is the subject of the next chapter.

• **the symptom**: aim the camera **up** at a tall building and its **parallel verticals converge** toward the top (keystoning); aim it down and they splay. Wide-angle shots show the same projective effect as exaggerated **near/far stretch** (the close part of a façade looms, the far part shrinks). Photographers find converging verticals **objectionable** in architecture — a building "looks more formal if the verticals stay vertical." [`fig-keystoning-cause`]

• **the cause** (the key reframing): this is **perspective projection doing its job**, not the lens misbehaving. Projection **preserves straight lines** but **does not preserve parallelism, angles, or lengths** — so a bundle of world-parallel verticals images as a bundle of lines **converging to a vanishing point**. (Contrast with *radial lens distortion* — barrel/pincushion — which **bends straight lines** and *is* an optical defect; that's a **different correction**, now developed in [[Optics, lenses, and aberration correction]] (*Aberrations correction* — radial distortion correction). Keystoning bends nothing; lines stay straight, they just stop being parallel.)

• **faces in a wide-angle shot — the perceptual stake**: the same near/far stretch is what makes **faces** shot up close with a wide lens look wrong — the nose looms, the ears recede. There's no single plane to rectify (a face is 3-D), so the fix is **per-face / content-aware un-distortion** — locally re-warp each face region toward how it would have looked from farther away. What makes this worth doing is *perceptual*, not merely geometric: a face's perceived **shape — and even its perceived personality** (trustworthiness, competence, attractiveness) — **depends on the camera-to-subject distance**, which is the basis for correcting wide-angle portrait distortion (Perona 2007; perceived-trait-vs-distance study Bryan et al. PMC4114730). This is the same focal-length-vs-distance fact that governs portrait lenses — cross-ref [[Fundamentals]] (Lenses and focal length).

• **the giveaway**: if the **image plane is parallel to the subject plane** (sensor parallel to the façade), then world-parallel lines parallel to the sensor **stay parallel in the image** — no convergence. Convergence appears exactly when you **tilt** the sensor off the façade. So the distortion is a function of **camera orientation**, and that is precisely what a homography can undo.

• **the idea**: the converging-verticals photo *is* a perspective image of a real planar rectangle (the façade). A **homography** maps between any two images of a plane — so there exists a single $3\times3$ projective warp $H$ that re-renders the façade **as if the sensor had been parallel to it**, sending the converging lines back to parallel. This is the **projective rung** of the DOF ladder from [[Warping and resampling]], applied here with the *target* being a fronto-parallel rectangle. [`fig-rectify-before-after`]

• **why one homography suffices (for a plane)**: reprojecting a plane = "project, rotate the camera, reproject," which is exactly a homography (depth drops out — *all points on the plane*, indeed all points along each ray, reproject identically). This is the same "depth doesn't matter for a planar-or-rotation warp" fact that makes panorama stitching work. **Cross-ref [[Manual panorama stitching from multiple views]]** — *identical machinery*; there the second image is another *view*, here it's the *target rectangle*.

• **solving $H$ — method 1, four corner correspondences**: pick **4 points** on the building (the corners of a known-rectangular feature — a window, the façade outline) and specify where they should go (a true rectangle, verticals vertical). 4 correspondences → 8 equations → solve the 8-DOF homography (up to scale) by **least squares / SVD**, exactly as in [[Manual panorama stitching from multiple views]]. This is the manual **perspective crop** in Photoshop. [`fig-rectify-before-after`, the 4 points]

• **solving $H$ — method 2, automatic via vanishing points**: detect bundles of **parallel lines** (the verticals, and the horizontals) — each images as a bundle meeting at a **vanishing point**. Choose the homography that sends the relevant vanishing point **to infinity** (making the verticals vertical/parallel again, and optionally the horizontals too). This needs **no user clicks** — it's the basis of Lightroom's **Upright** (it finds straight lines and squares them up) and of single-view metric rectification (Liebowitz & Zisserman 1998; Criminisi 2000). [`fig-vanishing-point-autosolve`]

• **applying it**: once $H$ is known, **inverse-warp and resample** — $\text{out}(\mathbf x)=\text{in}(H^{-1}\mathbf x)$, loop over output pixels, push back through $H^{-1}$, resample the input (the primitive from [[Warping and resampling]]). Same forward-vs-inverse and resampling care as everywhere.

• **in practice (tools)**: **Lightroom Transform / "Upright"** (auto and guided modes), **Photoshop perspective crop**, and dedicated architecture tools all do exactly this homography. The user experience is "drag the building straight"; the engine is the 4-point or vanishing-point homography.

• **fix it in the lens, not in post**: a **view camera** or **tilt-shift lens** corrects perspective **at capture**. The lens projects an image circle **larger than the sensor**; **shifting** the lens (or the sensor / back) selects an **off-centre** part of that circle while keeping the **sensor parallel to the façade**. Because the sensor stays parallel to the subject plane, **verticals never converge** in the first place — no post-warp, no resampling penalty. [`fig-tilt-shift-scheimpflug`]

• **equivalent to "wider-angle + crop"**: a shift is geometrically the same as taking a **wider, centred** view and **cropping** to the off-axis region you want — which is also why software "perspective crop" can approximate it (at the cost of resolution). **Tilt** (the other view-camera movement) instead controls the **plane of focus** via the **Scheimpflug** condition (lens, sensor, and subject planes meet in a line) — used to get a slanted plane all-in-focus, or, reversed, the "miniature/tilt-shift" toy-town look. **Cross-ref Optics — Special optics** for the optical derivation; here we only note it as the at-capture counterpart of the homography. The two are related by — unsurprisingly — **a homography** (the shifted-sensor image and the tilted-sensor image of the same façade differ by a projective warp).

• **trade-off** (when to use which): optical correction **preserves resolution and avoids resampling artefacts** but needs special (expensive, manual) gear and must be decided **at capture**; the software homography is **free and deferrable** ("decide later") but **resamples** (next).

• **rectification resamples and stretches**: making the far/top of the façade as wide as the near edge means the homography **enlarges** some regions and **compresses** others → the output has **non-uniform resolution** (top corners enlarged and **softened**; other parts compressed). The enlarged regions are interpolation-limited (no new detail — cross-ref [[Super-resolution and image priors]]); the compressed regions must be **prefiltered before downsampling** (**L16**, see the recurrence box) or they alias. The kernel footprint should follow the **local Jacobian** of $H$. [`fig-rectify-resample-stretch`]

• **you lose frame / pixels**: the rectified rectangle generally doesn't fill the original frame — corners read **outside** the input (boundary policy from [[Warping and resampling]]) and you usually **crop** to the valid region, sacrificing field of view and resolution. "Upright" with auto-crop is managing exactly this.

• **only a plane is exact** (the fundamental limit): a single homography exactly rectifies a **single plane**. A real 3-D scene (a building with depth, foreground objects, a receding street) has **multiple planes at different depths** — one homography can square up **one** of them, but everything off that plane is left distorted (and parallax between depths can't be removed by any 2-D warp). Genuinely correcting a 3-D scene needs **depth or multiple views** (image-based rendering / multi-view) — cross-ref view morphing and the panorama parallax caveat in [[Manual panorama stitching from multiple views]]. The honest scope of this chapter: **planar** perspective correction.

• **the single instance of the part spine (L17)**: this is the **simplest** member of the *establish a coordinate map, then transport the pixels along it* family — the map is **parametric** (one homography), the scene is a **single plane**, the input is **one image**. The map is *solved* (4 corners / vanishing points), not estimated.

• **closing link / where the family goes harder**: the next time the **same** homography machinery appears it aligns **two views** into a panorama ([[Manual panorama stitching from multiple views]], where the map must be *estimated* from feature matches); after that the maps become **dense and estimated** (optical flow) rather than parametric and solved. Same two steps throughout; only the map gets harder.

• **the two regimes** of correspondence (the taxonomy that organises this whole part): **dense optical flow** gives a 2-D motion vector at *every* pixel, but typically only between *two successive frames*; **feature tracking** gives motion at only a *sparse* set of points, but *threads them across a long sequence* of frames. Dense-but-short vs sparse-but-long. [`fig-track-vs-flow`]

• **why go sparse** (the case for tracking, from the slides' "motivations for sparse"): **scalability** — far fewer unknowns (a few hundred points, not a million pixels), which mattered enormously historically and still matters for real-time SLAM and on-device tracking; **robustness / conditioning** — *some pixels are genuinely easier to track than others*, so spend effort only where the answer is reliable (a corner), and refuse to guess in flat regions or along edges where it is hopeless (the aperture problem). Tracking is dense flow's *quality-controlled* sibling: it only reports where it is confident.

• **the Lagrangian framing** (forward-ref, made precise in [[Motion blur and temporal sampling]]): tracking **follows a particle** — "this *physical* point, where is it now?" — the **Lagrangian** view. Dense per-pixel flow leans **Eulerian** — "at this *fixed* pixel, what moved through?" This is the organising dichotomy for the part; tracking is its cleanest Lagrangian instance. (The slides note matching-based flow "becomes more Lagrangian" precisely by *following the patch* rather than differentiating at a fixed pixel.)

• **where tracked points go next**: the sparse trajectories are the *input* to structure-from-motion, SLAM, video stabilization, and bundle adjustment — wherever you need *the same physical points* identified over time, not a per-pixel flow. (Cross-ref [[Automatic panorama stitching from multiple views and feature matching]] — features → 3D.)

• **the core move**: take the **Lucas–Kanade** patch solve from [[Optical flow]] and apply it *to one small window around a feature*, then *iterate*, then *repeat for the next frame*. That is the whole Kanade–Lucas–Tomasi (KLT) tracker. (Lucas & Kanade 1981 = the registration update; Tomasi & Kanade 1991 = packaging it as a feature *tracker*; Shi & Tomasi 1994 = *which* features.)

• **the per-patch system** (reuse, don't re-derive — it is LK restricted to a window $W$): assume every pixel in $W$ shares one displacement $\mathbf{u}=(u,v)$ and stack the brightness-constancy constraint $I_x u + I_y v + I_t = 0$ over the window. The least-squares normal equations are

$$M\,\Delta\mathbf{u}=\mathbf{b},\qquad M=\sum_{\mathbf{x}\in W}\begin{pmatrix}I_x^2 & I_xI_y\\ I_xI_y & I_y^2\end{pmatrix},\qquad \mathbf{b}=-\sum_{\mathbf{x}\in W} I_t\begin{pmatrix}I_x\\ I_y\end{pmatrix},$$

so the update is $\Delta\mathbf{u}=M^{-1}\mathbf{b}$. **That $M$ is exactly the structure tensor** of Harris (the second-moment matrix) — *the same matrix, no new math*. (Slides: "per-patch linear system… can be seen as per-patch 1-step Horn–Schunck with strong constant smoothness.") [`fig-klt-is-harris`]

• **iterate, because LK is only a linearisation**: a single solve assumes sub-pixel motion (brightness constancy was Taylor-linearised). So **warp** the second-frame patch by the current estimate, recompute $I_t$ on the warped patch, re-solve for a residual $\Delta\mathbf{u}$, and **add** it — repeat until convergence. This is Gauss–Newton on the patch SSD (Baker & Matthews 2004's account).

• **coarse-to-fine for large motion**: LK assumes motion of roughly $\lesssim 1$ pixel. For real videos, run the iteration **down a pyramid** (Bouguet 2000): solve on a heavily-blurred/downsampled level where the displacement *is* small, **propagate** the estimate as the initial warp for the next-finer level, refine. Same coarse-to-fine logic as dense flow, localised to each feature. [`fig-klt-pyramid-iteration`]

• **across time**: having localised the feature in frame $t{+}1$, use that as the start for frame $t{+}2$, and so on. KLT is this $2\times2$ solve threaded greedily down the whole clip — cheap (one tiny matrix inverse per feature per frame) but **greedy and memoryless**, which is exactly what makes drift and occlusion bite (below).

• **pseudocode**: a short KLT loop (per feature: iterate warp-recompute-$I_t$ → build $M,\mathbf b$ → $\Delta\mathbf u=M^{-1}\mathbf b$ → add; then carry $\mathbf u$ to the next frame) shows the three nested layers — converge, then thread over time.

• 💡 **the hinge of the chapter**: the update $\Delta\mathbf{u}=M^{-1}\mathbf{b}$ is only meaningful when **$M$ is invertible and well-conditioned**. And $M$'s conditioning is read off its **eigenvalues** $\lambda_1\ge\lambda_2$ — the *same* eigenvalues that classify a patch as flat / edge / corner in Harris. So "*is this a good feature to track?*" is literally "*is the tracking matrix well-conditioned?*" which is *literally* "*is this a corner?*" — one question wearing three hats. [`fig-klt-is-harris`, `fig-good-features-to-track`]

• **the three cases** (reused from the Harris derivation, now read as *trackability*):

• **flat region** ($\lambda_1\approx\lambda_2\approx 0$): $M$ is near-zero, $M^{-1}$ blows up — no texture, motion totally ambiguous. **Untrackable.**

• **edge** ($\lambda_1\gg\lambda_2\approx 0$): $M$ is rank-deficient — motion *along* the edge is unconstrained (the **aperture problem**). You can recover only the across-edge component. **Half-trackable** — and the unstable half drifts.

• **corner** ($\lambda_1,\lambda_2$ both large): $M$ is well-conditioned, motion fully determined in both directions. **Trackable.**

• **Shi–Tomasi "good features to track"** (1994): pick features by the simple, principled test

$$\min(\lambda_1,\lambda_2) > \lambda_{\min},$$

i.e. require the *smaller* eigenvalue to be large enough — *both* directions well-constrained. This is the conditioning of $M$ stated directly; it is the same $\min(\lambda_1,\lambda_2)$ Shi–Tomasi corner response cross-referenced in [[Automatic panorama stitching from multiple views and feature matching]]. The Harris response $R=\det M - k(\operatorname{tr}M)^2$ is a (determinant-trace) proxy for the same thing. **The detector and the tracker agree by construction** — Shi & Tomasi derived "what is a good feature to *detect*" *from* "what is a good feature to *track*", not the other way round. [`fig-good-features-to-track`]

• **practical recipe**: compute $M$ over a window at every pixel (image gradients → products → Gaussian-weighted sum, exactly the Harris pipeline), score by $\min(\lambda_1,\lambda_2)$, **non-max suppress**, keep the top-$N$ well-spread points — these are the features you then KLT-track.

• **the window-size tension** (Shi & Tomasi, slide "Rest of the story"): a *small* window localises precisely but sees little texture (poorly conditioned $M$) and is fooled by noise; a *large* window is better conditioned but blurs across **multiple motions** (a single $\mathbf{u}$ no longer fits — depth discontinuities, object boundaries). The sweet spot trades localisation against conditioning — the recurring small-aperture-vs-large-aperture tension of all patch methods.

• **drift** — the cost of greediness: each frame's tiny LK residual is imperfect; chained over hundreds of frames the errors **accumulate** and the window slowly slides off the true feature (sub-pixel bias becomes pixels). The classic fix is to **anchor to the first frame**: track against the *original* template (frame $t_0$), not the previous frame, so error doesn't compound — but then **appearance change** fights you. [`fig-track-drift-occlusion`(a)]

• **appearance change** — the template goes stale: over a long clip the patch changes due to **lighting**, **out-of-plane rotation**, **scale** (the object nears/recedes), and perspective. A pure-translation template match degrades. Shi & Tomasi's fix: track translation frame-to-frame (small motion, cheap) but **verify against the anchor template with an *affine* warp** — allow the patch to rotate/scale/shear when checking quality. The affine **dissimilarity**

$$\varepsilon=\sum_{\mathbf{x}\in W}\big[\,I_2(A\mathbf{x}+\mathbf{d})-I_1(\mathbf{x})\,\big]^2$$

measures how well the current patch still matches its origin under the best affine warp $A$; a large $\varepsilon$ means the feature is no longer itself.

• **occlusion** — the point disappears: when a feature passes **behind** another object (or off-frame, or into a shadow/specular flip), there is *no* correct match — but a greedy LK solve will happily lock onto *whatever* nearby texture minimises SSD, silently corrupting the track. **Detection: the dissimilarity $\varepsilon$ spikes** → declare the feature **lost** and drop it rather than trust a garbage match. (Slides "Issue: Motion Boundaries — no matches can be found at motion boundaries… also at frame boundary.") [`fig-track-drift-occlusion`(b)]

• **re-detection** — keep the set populated: as features are lost to drift/occlusion/leaving-frame, periodically **re-run Shi–Tomasi** and spawn fresh features in the gaps, so the tracked set stays dense enough and well-distributed. Detection and tracking become an interleaved loop (slides: "Combine tracking and detection; keep refining detection model over time"). [`fig-track-drift-occlusion`(c)]

• **the honest limitation** that motivates the next section: classical KLT is **per-point, greedy, and forgetful** — each feature is tracked independently with no memory of where it went and no way to *recover* a point after it reappears from behind an occluder. Points also **drift relative to each other** (no global consistency). These are exactly the failures learned trackers were built to fix.

• **the shift** (the *bitter-lesson* successor to KLT; **L8**): keep the task — "track this physical point through a video, even through occlusion" — but replace the hand-built brightness patch with **learned features** and the greedy per-frame solve with a **learned iterative update**, trained on synthetic video with ground-truth tracks (FlyingThings3D, Kubric, etc.). The classical scaffolding survives *as architecture*: a 4-D **cost volume** (appearance similarity), **iterative refinement** of the trajectory, and now an explicit **visibility / occlusion** prediction. [`fig-modern-point-trackers`]

• **PIPs — Particle Video Revisited** (Harley et al. 2022): tracks **one point at a time** over a temporal window; solves jointly for its $(x,y)$ location *and* an appearance vector across all frames, **predicts visibility per frame**, and **initialises with a constant** location/appearance then iteratively refines (RAFT-style). The reintroduction of *long-range, occlusion-aware* particle tracking that KLT lacked.

• **TAP-Vid** (Doersch et al. 2022): frames the modern task — **Track Any Point** — and supplies the **benchmark** that made progress measurable (the long-standing bottleneck, as in optical flow, is **ground-truth data**).

• **CoTracker — track points *together*** (Karaev et al. 2023): the key insight, stated as a slogan, is that **tracking points jointly beats tracking them independently** — points on the same object constrain each other, giving **global consistency** and far better **occlusion handling** (a point hidden behind an object is inferred from its visible neighbours). Mechanism: a **transformer** over a sliding window with **factorised attention across space, time, and the group of tracks**, iteratively updating each track's position/appearance/visibility — plus extra **register tokens** so the network has scratch space for global info (otherwise it hijacks real pixels, causing spiky artifacts).

• **why this closes the section**: every failure of greedy KLT — drift, single-point myopia, silent occlusion corruption — is answered by *learn the features, refine iteratively, track together, predict visibility*. The structure-tensor "is this point trackable?" question is now answered implicitly by the learned features and the joint optimisation, but the **problem** is identical to the one Lucas, Kanade, and Tomasi posed. (Full model details: [[Deep learning]]; the same *neuralise-the-classical-iteration* pattern is RAFT in [[Optical flow]].)

• **definition**: the **2-D motion vector of every pixel** between two (usually consecutive) frames — a dense field $(u(x,y),\,v(x,y))$. Visualized with a **color key**: hue = motion direction, saturation = speed (Baker et al. 2011). [`fig-flow-field-colorkey`]

• **why it matters (downstream)**: frame interpolation / slow-motion, video super-resolution, in-painting, stabilization, special effects; action recognition, tracking, surveillance; egomotion / SLAM; **video compression** (motion-compensated prediction); medical registration; even optical mice and fluid-flow physics. A workhorse primitive.

• **relation to stereo**: disparity is optical flow **specialized to a static scene with a known camera baseline** — the search collapses to the **1-D epipolar line**, so there's a **single unknown (depth/disparity) per pixel** instead of two. Flow is the unconstrained 2-D cousin (cross-ref [[Image alignment]] / multi-view).

• **honesty up front**: optical flow is **ill-defined** in places — flat untextured regions (no information), occlusion / motion boundaries (a pixel *has* no correspondence), lighting changes and specularities (brightness *isn't* constant). Worth keeping in mind that "the flow" is an idealization that the math itself will warn us is sometimes unrecoverable.

• **the progression of "what is a match"** (foreshadow the chapter and its sequels): per-pixel **RGB + brightness constancy + gradient** (this chapter) → **window/patch** matching (SSD, cost volume) → hand-designed **features** (SIFT/HOG) → **learned** features (RAFT and successors). Each step buys robustness at the cost of locality.

• **the assumption (brightness / color constancy)**: a scene point keeps the same intensity as it moves over one timestep — it just *relocates*:

$$I(x,\,y,\,t) = I(x+u,\;y+v,\;t+1).$$

($u,v$ = the per-pixel displacement we want.) This is the "color constancy assumption" of Horn & Schunck 1981. [`fig-brightness-constancy`]

• **linearize (first-order Taylor)**: expand the right side about $(x,y,t)$, assuming small motion:

$$I(x+u,y+v,t+1) \approx I(x,y,t) + I_x\,u + I_y\,v + I_t.$$

Subtract $I(x,y,t)$ from both sides of brightness constancy → the **optical-flow constraint equation (OFCE)**:

$$\boxed{\,I_x\,u + I_y\,v + I_t = 0\,}\qquad\text{equivalently}\qquad \nabla I\cdot(u,v) + I_t = 0,$$

with $I_x,I_y$ the **spatial** gradients and $I_t$ the **temporal** gradient (frame difference).

• 💡 **"only the gradient matters"**: motion is invisible in **flat** regions — where $\nabla I = 0$ the constraint says $I_t=0$ and tells you *nothing* about $(u,v)$. Information about motion lives **entirely in the image gradient**. (This is exactly the lever Eulerian video magnification pulls: amplify the *temporal* change $I_t$ in proportion to the spatial gradient.)

• **the constraint is a line, not a point**: one scalar equation in two unknowns. In the $(u,v)$ plane it's a **line**; the true flow lies *somewhere on it* but is otherwise undetermined. We can pin down only the component **along $\nabla I$**: the **normal flow** $\;u_\perp = -I_t / \lVert\nabla I\rVert$ (magnitude), directed along the gradient. The component **tangent to the edge is free**. [`fig-flow-constraint-line`]

• **the statement**: from a *single* pixel (one OFCE) you can recover **only the normal flow** — the motion **perpendicular to the local edge**. The motion **along** the edge is unobservable. *One equation, two unknowns.* [`fig-aperture-problem`]

• **the intuition (the "aperture")**: watch a moving straight edge through a small hole. A long diagonal bar sliding behind the aperture looks the same whether it moves horizontally, vertically, or perpendicular to itself — you cannot tell. Only motion that **changes what's inside the aperture** (the ⟂ component) is visible. The **barber-pole** illusion (stripes appear to move *up* though the pole spins sideways) is the same effect.

• **why it's not a bug**: the math is *honestly reporting* a genuine ambiguity — the data truly don't determine the tangential motion locally. So "solving" flow = **adding a second assumption** to supply the missing equation. The two classic choices:

• **a local one** — neighbors share the motion → **Lucas–Kanade**;

• **a global one** — the field is smooth → **Horn–Schunck**.

• **the perceptual flip side**: human vision suffers the aperture problem too (we mis-perceive these stimuli), and resolves it by **integrating across the object** (corners, line-endings) — exactly what the algorithms do by aggregating a window or imposing smoothness.

• **the assumption**: pixels in a **small window** $W$ all share the **same** motion $(u,v)$. Now we have **many** OFCEs (one per pixel in $W$) for the **two** unknowns — over-determined → **least squares** resolves the aperture problem *and* regularizes against noise. (Lucas & Kanade 1981.)

• **stack the equations**: with $n$ pixels in $W$, form

$$A = \begin{bmatrix} I_x^{(1)} & I_y^{(1)} \\ \vdots & \vdots \\ I_x^{(n)} & I_y^{(n)} \end{bmatrix},\quad b = \begin{bmatrix} I_t^{(1)} \\ \vdots \\ I_t^{(n)} \end{bmatrix},\qquad A\begin{bmatrix}u\\v\end{bmatrix} = -b.$$

The least-squares solution solves the **normal equations**:

$$\boxed{\,A^\top A \begin{bmatrix}u\\v\end{bmatrix} = -A^\top b\,},\qquad A^\top A = \sum_W \begin{bmatrix} I_x^2 & I_x I_y \\ I_x I_y & I_y^2 \end{bmatrix},\quad A^\top b = \sum_W \begin{bmatrix} I_x I_t \\ I_y I_t\end{bmatrix}.$$

• 💡 **the matrix $A^\top A$ is the Harris structure tensor** (the second-moment matrix). The very same object decides corner detection ([[Automatic panorama stitching from multiple views and feature matching]]) and flow conditioning — **flow is solvable exactly where Harris finds a corner**:

• **two large eigenvalues** (a **corner** / well-textured patch): $A^\top A$ invertible, full 2-D flow recoverable, well-conditioned. *These are the "good features to track."*

• **one large eigenvalue** (an **edge**): rank-deficient → only **normal flow** (the aperture problem, *re-derived* — along-edge motion is the null direction).

• **both small** (a **flat** region): no information; flow indeterminate.

This is the bridge from dense flow to **sparse feature tracking** (KLT): just track the pixels where $A^\top A$ is well-conditioned. [`fig-flow-corner-edge-flat`]

• **scope & relation to HS**: LK assumes **small motion** ($\sim$1 px, from the linearization) and a window large enough to be informative but small enough to be locally constant. It can be read as a **per-patch, one-step Horn–Schunck with a hard "constant flow" smoothness** — local and direct, vs HS's global and iterative. Extended to large motion via **coarse-to-fine pyramids** and iterative warping. LK is also the engine of **parametric image alignment** (register two images under one affine/homography — same gradient least-squares; cross-ref [[Image alignment]]). [`fig-LK-vs-HS`]

• **the assumption**: instead of constancy in a window, impose that the flow field is **globally smooth** — *neighboring pixels have similar motion*. This fills in the unconstrained (aperture / flat) directions from the data-rich ones, propagating motion into textureless regions. (Horn & Schunck 1981.)

• **the energy** (data-fit + prior — the classic inverse-problem shape):

$$E(u,v) = \iint \underbrace{\big(I_x\,u + I_y\,v + I_t\big)^2}_{\text{data: brightness constancy}} \;+\; \lambda\,\underbrace{\big(\lVert\nabla u\rVert^2 + \lVert\nabla v\rVert^2\big)}_{\text{prior: smoothness}}\;\,dx\,dy.$$

The **data term** is the OFCE residual (penalizes violating brightness constancy); the **smoothness term** penalizes spatial *variation* of the flow; $\lambda$ trades them off (large $\lambda$ → smoother, more diffused; small $\lambda$ → trusts the data, noisier).

• **it's convex / linear to solve**: quadratic in $(u,v)$ → the Euler–Lagrange equations are **linear** (a large sparse system coupling all pixels — a screened-Poisson-like diffusion). Solve by **iterative** updates (Gauss–Seidel / Jacobi): each pixel's flow relaxes toward its **neighborhood average** corrected by the local brightness constraint. Globally smooth, fills the apertures.

• **LK vs HS — the contrast to land**: **local window + least squares** (LK) vs **global field + smoothness prior** (HS); **one-step** vs **iterative**; **constant-in-a-patch** vs **smoothly-varying-everywhere**. Two ways to supply the aperture problem's missing equation — *aggregate locally* or *regularize globally*. [`fig-LK-vs-HS`]

• **modern classical flow** — the *Secrets* paper (Sun, Roth & Black 2010): the bones are still HS, but what actually wins is **robust** (non-quadratic, $L_1$/Charbonnier) penalties on both terms (so edges/occlusions don't over-smooth), **median filtering** of the flow between iterations, **coarse-to-fine pyramids**, and **edge-aware / non-local** smoothness. The lesson: the *energy is the design* — same skeleton, better data/prior terms.

• **the problem**: the Taylor linearization (hence both LK and HS) assumes **sub-pixel** motion. Real motion is many pixels — the linearization is invalid and the solver diverges or finds the wrong (aliased) match.

• **the fix — pyramids** (cross-ref [[Linear pyramids and wavelets]]): build a **Gaussian/Laplacian pyramid** of both frames. On the **coarsest** (heavily blurred & downsampled) level, even large motion is only a fraction of a pixel → the linear method works. Then, going finer:

1. **upsample** the coarse flow to the next level (and scale its magnitude);

2. **warp** frame 2 toward frame 1 by the current flow estimate ([[Warping and resampling]]);

3. estimate only the **residual** (now small) flow on the warped pair, and **add** it to the running estimate.

Repeat to the finest level. [`fig-coarse-to-fine-flow`]

• **why it's the right picture**: this is a **Laplacian-pyramid** view of flow — get the *low-frequency / large-scale* motion on blurry images, then add the *high-frequency* motion as a residual at each finer scale. The warp-then-refine loop is exactly the iterative scheme that **RAFT** and PWC-Net later learn.

• **pseudocode**: a coarse-to-fine block (loop levels: upsample-and-scale `flow` → `warp(frame2, flow)` → estimate residual `ΔLK` → add) makes the warp-then-refine loop explicit — the scheme RAFT later neuralizes.

• **a different attack — the cost volume**: rather than linearize, **match** patches directly. For each pixel in frame 1, score similarity (SSD / correlation) against candidate offsets in frame 2 → a **cost volume** (4-D: $(x_1,y_1)\times(x_2,y_2)$, or per-displacement). This keeps brightness constancy but **drops the linearization** (more "Lagrangian"), handling large motion at the cost of a big search — pruned by **coarse-to-fine** and (later) learned features. The cost volume is the centerpiece of learned flow.

• **the framing (L8)**: keep the classical *skeleton* — match (data term) + iterative update (with smoothness/context) — but **learn** the pieces instead of hand-designing them. (Forward-ref [[Machine learning]]: a learned operator swaps a hand-built prior/feature for one fit to data.)

• **RAFT** (Teed & Deng 2020, ECCV best paper) — the three learned ingredients on the classical scaffold: [`fig-raft-skeleton`]

1. **learned per-pixel features** replace raw RGB in the matching (robust to lighting, texture) — plus a separate **context** encoder of frame 1 for edge-aware regularization;

2. an **all-pairs 4-D correlation (cost) volume** — precompute similarity between every pair of pixels, pooled into a **pyramid** for multi-scale lookup (the cost-volume idea, made efficient);

3. a **recurrent update operator** (GRU, shared weights) that **iteratively refines** the flow field — at each step it **looks up** correlations at the current flow estimate and predicts an increment, exactly **mimicking an iterative optimizer** (the unrolled HS/coarse-to-fine loop), but with the update *learned*. All differentiable, trained end-to-end.

• **why flow got the CNN treatment late**: flow is **non-local** (a pixel can match anywhere) — hard for ordinary convolutions; the explicit **cost volume** is the inductive bias that makes non-local matching tractable. Training data is scarce (real flow ground truth is hard), so synthetic sets (FlyingChairs/Things, Sintel, KITTI) and **self-supervised warping losses** (SMURF) do heavy lifting.

• **the through-line to land**: RAFT is the classical pipeline **with its hand-crafted parts (features, the iterative minimizer) replaced by learned ones** — *scaffolding from traditional methods, weights from data*. It is **not** "bitter-lessoned" into a generic transformer: the cost-volume + recurrent-update **inductive bias still wins**, which is itself the lesson — non-local matching is hard enough that problem-specific structure still pays. (Lineage to note: FlowNet 2015, the first CNN flow; PWC-Net 2018, the *pyramid+warping+cost-volume* "Lucas–Kanade-flavored" net; transformer variants for the cost volume.)

• **closing hand-off**: with a dense, automatic correspondence field in hand, the **transport** tasks open up — warp/interpolate between frames for **slow-motion** ([[Frame interpolation and slow-motion synthesis]]), stabilize, compress, magnify motion ([[Morphing]] — motion magnification), or lift to **scene flow / SfM** (the 3-D cousins). *Correspondence first, then transport* — the part's spine.

• **global** tone mapping applies **one curve to every pixel** (a function of intensity alone — gamma, an S-curve, Reinhard's global operator). Simple, fast, monotone — but a single curve **cannot both compress the overall range and preserve local contrast**: flatten the range and local detail goes with it.

• **local** tone mapping lets the mapping **depend on a pixel's neighbourhood** — compress the *large-scale* range while *keeping* local contrast. The cost is the risk of **halos** (and reversals) where the locality is mishandled — the recurring failure mode introduced in BASIC.

• the recap point: every local operator below is a different answer to the same question — *how do you separate "overall brightness" (compress it) from "local detail" (keep it)?*

• **gradient-domain** (Fattal et al. 2002): attenuate **large gradients**, then **reconstruct** the image from the edited gradient field (a Poisson solve) — compress range by squashing big luminance steps while leaving small ones. Developed in [[Poisson image editing]].

• **bilateral base/detail** (Durand & Dorsey 2002): split the image into a **base** (large-scale, edge-preserving bilateral) and **detail**; **compress the base**, **keep the detail**, recombine. The canonical edge-aware local operator — developed in [[Edge-preserving techniques]].

• **local Laplacian** (Paris, Hasinoff & Kautz 2011): manipulate a **Laplacian pyramid** with a per-coefficient remapping that **avoids halos by construction** — local contrast control without the bilateral's artifacts. [[Edge-preserving techniques]].

• **exposure fusion** (Mertens et al. 2007): blend a **bracket of exposures** by per-pixel quality weights (well-exposedness, contrast, saturation) in a pyramid — *tone mapping without ever forming an HDR image.* (It straddles single/multi-image — flagged in the Known Issue.)

• **learned** (HDRnet, Gharbi 2017; FiveK retouching): **predict** a local (bilateral-grid affine) tone mapping from data — the operator itself learned from expert edits. [[Deep learning]].

• the table to land: same goal (separate and treat large-scale vs local), five mechanisms — gradient field, base/detail, pyramid remap, multi-exposure blend, learned. [fig-tonemap-taxonomy]

• tone mapping is **not** a digital invention — photographers have **locally controlled tone** since the wet darkroom. **Dodge** (hold light back from a region → lighter) and **burn** (add light → darker) are **manual local tone mapping** with the hands and bits of card under the enlarger.

• the **Zone System** (Ansel Adams): a disciplined method for **mapping scene luminances onto print tones** — meter the scene, place key values on chosen "zones," and develop/print to control where highlights and shadows land. It is **tone mapping as craft**: deciding the global transfer *and* the local adjustments before computation existed.

• the analog **characteristic curve** of film/paper (the H&D curve — toe, linear, shoulder) is the **global** tone curve in physical form; its shoulder is highlight roll-off, its toe is shadow compression — exactly what a digital global operator emulates.

• the bridge to land: digital global tone mapping = the **characteristic curve**; digital local tone mapping = **dodge & burn + the Zone System**, now automatic and per-pixel. The computation **mechanized** a century-old craft (cross-ref [[Human factors and the art of photography]]; the Intro's Adams "negative = score, print = performance" analogy).

• **goal**: produce a higher-resolution image than the camera measured — more pixels that are *meaningfully* sharper, not just upsampled. Contrast with plain **interpolation** (bicubic), which adds pixels but no new frequencies → soft.

• **the forward model**: a low-res measurement is a high-res scene $x$ **blurred** by the optics/PSF $k$, **downsampled** by factor $s$, plus **noise** $n$: $y = (k * x)\downarrow_s + n$. Exactly the linear forward model $y = Ax + n$ of [[Linear Inverse Problems and Regression]], with $A = (\text{downsample})\cdot(\text{blur})$. [`fig-sr-forward-model`]

• **why it's ill-posed**: downsampling is **many-to-one** — it discards the high frequencies above the new Nyquist limit. Infinitely many high-res $x$ pass through the *same* $y$ (add any detail that the blur+downsample would have annihilated and $y$ is unchanged). So $A$ has no usable inverse; the data **under-determines** the answer. [`fig-sr-ill-posed`]

• **the resolution — a prior**: recover with **data-fit + regularizer**, $\hat{x} = \arg\min_x \tfrac12\lVert (k*x)\downarrow_s - y\rVert^2 + \lambda\,\Phi(x)$. The data term keeps $\hat{x}$ consistent with what was measured; $\Phi(x)$ (the **prior**) picks, among all consistent $x$, the one that *looks like a real image*. This is the MAP reading: $\Phi = -\log p(x)$, a prior × likelihood (Refreshers — Bayes).

• 💡 **the throughline**: without $\Phi$ the problem has no answer — see the **L10 big-lesson box** above. Everything below is *which* prior and *how* it's applied.

• **intuition-first framing**: SR is not "magnifying glass for files." It is "guess the most plausible scene that would have produced this blurry, aliased, noisy little image" — and *plausible* is defined by the prior.

• **classic single-image SR**: one frame in, one bigger frame out. The data fixes nothing new above Nyquist → **all** the extra detail comes from the prior. Two classic priors: **internal** (patches recur across scales within the *same* image — Glasner 2009) and **external/example-based** (a database / learned model of how low-res patches map to high-res — Freeman 2002). Pure single-image SR is therefore **hallucination-leaning** by necessity. [`fig-sr-recon-vs-halluc`, right panel]

• **multi-frame / burst SR** (Wronski 2019, the Pixel "Super Res Zoom"): capture a **burst** of frames; **hand tremor** shifts the scene by **sub-pixel** amounts between frames. Each frame samples the *same* continuous scene at slightly different grid offsets → align with sub-pixel **registration** and fuse → a **denser** effective sampling grid. This is **genuine information**: aliasing (normally a defect) becomes the signal, because the shifts let you separate frequencies a single frame folded together. No dedicated prior needed for the *recovered* detail — the frames supply it (though robust fusion still needs a noise/motion model). [`fig-burst-subpixel`]

• key sub-points: **sub-pixel alignment is the whole game** (mis-registration → blur/ghosting); **robust merge** rejects moving objects / occlusion (else ghosts); replaces the demosaic step too (the color mosaic's offsets help). Ties to BASIC — sampling/aliasing and to multi-frame denoising ($1/\sqrt{N}$ averaging).

• **hybrid space–time SR**: trade **temporal** resolution for **spatial** (or vice versa) — combine several low-spatial/high-temporal captures (or a fast + a slow camera) so motion provides extra spatial samples and vice versa. Same idea as burst, generalized to the time axis.

• **the organizing axis** — *how much is measured vs invented*: many sub-pixel-shifted frames ⇒ mostly **reconstruction**; a single frame ⇒ mostly **hallucination**; real systems sit in between. This axis is the next section. [`fig-sr-recon-vs-halluc`]

• **the honest distinction**, and why it matters for a reader: both make a sharper picture, but only one adds **truth**.

• **reconstruction-based**: the extra detail was **genuinely measured** — by multiple sub-pixel-shifted frames (burst), or by deconvolving frequencies the optics merely attenuated (not killed). It is *recovered*. Verifiable, repeatable, faithful.

• **hallucination-based**: a **learned prior** synthesizes detail that is *plausible* for natural images but was **never in the measurement** — pores on a face, text on a distant sign. It can look stunning and be **wrong** (invent the wrong digits, the wrong texture). [`fig-sr-recon-vs-halluc`]

• **the learned hallucination models** (used here as **priors**, treated as *models* in [[Deep learning]]): **Real-ESRGAN** (Wang 2021 — GAN-trained on a realistic degradation pipeline, sharp/textured outputs) and **SwinIR** (Liang 2021 — transformer restoration). They learn $p(\text{high-res} \mid \text{low-res})$ from data and sample a plausible completion. **Forward-ref**: their architecture and training live in the ML chapter; here they're one end of the prior axis.

• **caveats to surface** (pedagogical honesty): metric trap — a hallucinated result can score *better* on PSNR/SSIM yet be unfaithful (and a faithful one can score worse); the **perception–distortion tradeoff**. Forensic/ethical note: SR'd faces and license plates are **not evidence**. (Cross-ref learned perceptual metrics — LPIPS — in [[Deep learning]].)

• **resolve the open "in ML at this point?" question** — **verdict: keep super-resolution HERE**, as the *image-prior* story that unifies the whole single-image part. The chapter owns the **prior** abstraction (ill-posedness → regularizer → denoiser-as-prior → diffusion). The **learned models** themselves (Real-ESRGAN, SwinIR, GANs, diffusion nets) are taught as models in [[Deep learning]] / [[Generative AI and diffusion]], cross-referenced — not duplicated. So: priors here, models there.

• **the unifying move**: in $\hat{x} = \arg\min_x \tfrac12\lVert Ax - y\rVert^2 + \lambda\Phi(x)$, splitting algorithms (**ADMM / HQS**) separate the two terms into alternating steps. The regularizer step turns out to be **exactly a denoising operation** (the proximal operator of $\Phi$ = "find the nearby image the prior likes" = denoise). [`fig-pnp-loop`]

• **Plug-and-Play (PnP) priors** (Venkatakrishnan 2013): so **drop in *any* denoiser** $\mathcal{D}_\sigma$ for that step — even one with **no closed-form prior** (BM3D, NL-means, a learned CNN). Alternate:

• **data-fit step**: $z \leftarrow \arg\min_x \tfrac12\lVert (k*x)\downarrow_s - y\rVert^2 + \tfrac{\rho}{2}\lVert x - \tilde{x}\rVert^2$ (enforce the measurement; a linear solve — CG/FFT, per [[Linear Inverse Problems and Regression]]);

• **denoise step**: $\tilde{x} \leftarrow \mathcal{D}_\sigma(z)$ (impose the prior — *any* denoiser).

• payoff: **decouples** the imaging physics (the forward model $A$, swap per task — SR, deblur, inpaint, demosaic) from the **image prior** (the denoiser). One good denoiser ⇒ a solver for *every* inverse problem. (FlexISP, Heide 2014, builds a whole ISP this way.) [`fig-denoiser-as-prior-spectrum`]

• **RED (Regularization by Denoising)** (Romano 2017): make the prior **explicit** from a denoiser. Define $\Phi(x) = \tfrac12 x^\top(x - \mathcal{D}(x))$; under mild conditions its **gradient is just the denoising residual**, $\nabla\Phi(x) = x - \mathcal{D}(x)$. Then plain **gradient steps** minimize the regularized objective — the denoiser defines an actual energy, not only an opaque proximal step. (PnP ≈ proximal/implicit; RED ≈ explicit-gradient.)

• **why "universal"** (the section's headline): the **denoiser is the prior**. Improve the denoiser and *every* recovery task improves for free. This reframes "image prior" from a hand-derived penalty (TV, sparsity) to "whatever your best denoiser knows." [`fig-denoiser-as-prior-spectrum` — the spectrum from TV → BM3D → learned → diffusion]

• 💡 **callback to L10**: PnP/RED is the *mechanism* by which the not-optional prior enters — the denoise step IS the prior step.

• **Tweedie's formula**: for a Gaussian-noised observation $z = x + \sigma\epsilon$, the **MMSE denoiser** (posterior mean) is $\hat{x} = z + \sigma^2\,\nabla_z \log p_\sigma(z)$. So the **score** $\nabla_z \log p_\sigma(z)$ and a **Gaussian denoiser** are the *same object* (one is the other rearranged). [`fig-denoiser-as-prior-spectrum`, right end]

• **therefore diffusion = iterated denoising**: a diffusion model learns the score at every noise level; **sampling** repeatedly applies "denoise a little, add a little noise, repeat" from pure noise down to a clean image. That outer loop is precisely **PnP/RED taken to the continuous limit** — the prior-step denoiser is the learned score, run over a noise-level schedule.

• **consequence for recovery**: conditioning diffusion sampling on a measurement ($y$ via the forward model $A$) = a PnP/RED solver whose prior is the strongest learned denoiser we have → state-of-the-art SR / deblur / inpaint. This is the bridge from this chapter's *prior* story to generative models.

• **forward-ref**: the full diffusion treatment (DDPM, latent diffusion, score-matching, samplers) is in [[Generative AI and diffusion]]; here we only establish the **identity** denoiser ≡ score ≡ diffusion-step, closing the prior throughline. The diffusion chapter cross-refs back to *Denoising as a universal prior*.

• **closing idea** (hand-off to [[#Blind deblurring]]): the same "data-fit + prior, and the prior is doing the heavy lifting" story now plays out where the **forward operator itself is unknown** — blind deblurring — and where hand-built priors (dark-channel) still earn their keep.

• the trap: with a *known* kernel and *no* noise, deconvolution is exact — divide by the kernel in Fourier, done. Add even a *whisper* of noise and that exact inverse is **useless**. [fig-naive-deconv-noise]

• where it breaks, intuitively: blur is a **low-pass** filter — it *kills* high frequencies (the kernel's frequency response $\hat K$ goes to ~0 there). The inverse filter $1/\hat K$ therefore **divides by almost zero** at exactly those frequencies. The signal there is gone, but the *noise* isn't — so you amplify noise by huge factors. Result: an image of pure speckle, not a sharp photo.

• the linear-algebra twin: same statement as **small singular values** of the blur matrix. Inversion = dividing each component by its singular value; the tiny ones (high frequencies) are where noise explodes. Naive inversion has **no defense** because nothing tells it "this frequency is noise, not signal."

• 💡 **Big lesson (recurrence of L10):** *real measurements have noise, and inversion amplifies it — so naive inversion fails and a prior is not optional.* The forward problem (blur, project, downsample) is stable and loses information; running it backward is **ill-posed** and **noise-amplifying**. You cannot recover what the forward model destroyed — you can only **fill it in with prior knowledge.** This is the spine of the whole chapter (and of super-resolution, denoising, inpainting). (→ see Big lesson **L10** · the prior is not optional; first appears in [[#Super-resolution and image priors]].)

• the fix, stated: don't invert blindly — **trade off** fidelity to the measurement against plausibility of the answer. Where the kernel is strong (low freq), trust the data; where it's weak (high freq, noise-dominated), back off and rely on what we *expect* images to look like. That trade-off, optimal under a Gaussian-noise / known-spectrum assumption, **is the Wiener filter** (next).

• encoding note: deconvolution assumes the blur is **linear in scene radiance**, so do it in **linear light** (un-gamma first), per L2 — gamma/log distorts the additive convolution model. State this assumption up front.

• one-line idea: take the naive inverse filter and **attenuate it wherever noise would dominate**, frequency by frequency. The amount of attenuation is governed by the **signal-to-noise ratio** at each frequency.

• the formula (per frequency, in Fourier): $\hat X = \dfrac{\overline{K}\,Y}{|K|^2 + \mathrm{SNR}^{-1}}$. Read it: the $\overline K/|K|^2$ part *is* the inverse filter; the $+\,\mathrm{SNR}^{-1}$ in the denominator is the **regularizer** that stops the blow-up. Equivalently the inverse filter times a gain $|K|^2/(|K|^2 + S_n/S_x)$ (noise spectrum $S_n$, signal spectrum $S_x$). [fig-wiener-tradeoff]

• limit checks (hand-verifiable): **high SNR** ($\mathrm{SNR}^{-1}\!\to 0$) → recovers the exact inverse filter $1/\hat K$ (no noise, invert fully). **Where the kernel vanishes** ($|K|\to 0$) → the $+\,\mathrm{SNR}^{-1}$ keeps the denominator finite, so the gain → 0 instead of ∞: those lost frequencies are **left alone**, not amplified. Exactly the desired behavior.

• where it comes from (just the intuition, no derivation): the **MMSE** linear estimator under Gaussian noise — minimize expected squared error between estimate and truth → this filter. The "$\mathrm{SNR}^{-1}$" is literally a **prior on the image's power spectrum** ($1/f$-ish for natural images). So even the classic Wiener filter is **already a prior-driven method** — that's the bridge to everything that follows.

• limitation → the sequel: Wiener assumes a **known kernel** and a **stationary Gaussian** prior (a power-spectrum prior). Real images aren't Gaussian — they have **edges**, and the right prior is **sparse gradients**, not a power spectrum. Swapping the Gaussian prior for a sparse-gradient one gives the modern non-blind deconvolution (Levin 2007; Krishnan & Fergus 2009) — and when even the *kernel* is unknown, we get blind deblurring (next).

• forward-ref: the general version "any denoiser as the prior" is **Plug-and-Play / RED** in [[Super-resolution and image priors]]; the *learned* prior is [[Machine learning]].

• the new difficulty: now we **don't know the blur kernel** $K$ either (camera shake gives an unknown, photo-specific PSF). We must recover **both** $X$ and $K$ from one blurry $Y$ — twice as many unknowns as equations.

• the chicken-and-egg: knowing the sharp image makes the kernel easy (divide), and knowing the kernel makes the image easy (non-blind deconv) — but we have **neither**. [fig-blind-chicken-egg]

• why it's genuinely ill-posed (the trap to call out): the blurry image **explains itself** — $Y = (\text{sharp }X) * (\text{real blur }K)$ fits, but so does $Y = (\text{the blurry image}) * (\delta)$, i.e. "no blur at all." A naive joint optimizer happily returns the **no-op / delta-kernel** solution because it fits the data perfectly. Data-fit alone is hopeless.

• what breaks the tie — **natural-image priors:** real sharp photos have **sparse, heavy-tailed gradients** (mostly flat regions, a few strong edges); **blurring spreads** those strong edges into many weak gradients, *raising* the prior's cost. So the prior **prefers the sharp explanation** — the delta-kernel "no-blur" answer is penalized because the *blurry* image has the wrong (too-dense, too-Gaussian) gradient statistics. [fig-natural-image-gradient-prior]

• the objective: $(\hat X,\hat K) = \arg\min_{X,K}\ \lVert K*X - Y\rVert^2 + \lambda\,\rho(\nabla X) + \mu\,\psi(K)$, with $\rho$ a **sparse** penalty ($\lVert\nabla X\rVert_\alpha$, $\alpha<1$, a hyper-Laplacian) and $\psi$ a non-negativity / sparsity / energy constraint on the kernel. Non-convex in $(X,K)$ jointly.

• two ways people solve it:

• **alternating minimization** (the obvious approach): fix $X$, solve for $K$; fix $K$, do non-blind deconv for $X$; repeat — usually **coarse-to-fine** (estimate the kernel on a downsampled image first, where it's small, then refine). Simple but can collapse to the trivial solution if the prior isn't handled carefully.

• **pseudocode block** (in the draft): the coarse-to-fine alternation — `K ← δ; for each pyramid level: repeat { X ← deconv given K; K ← fit given X }; final non-blind deconv`.

• **the Fergus 2006 insight / Levin 2009 lesson:** don't MAP-estimate $X$ and $K$ *jointly* — that **biases toward the blurry (no-blur) solution**. Instead **marginalize over the image** and estimate the **kernel alone** (variational Bayes), *then* do a single non-blind deconvolution. Levin et al. 2009's evaluation is the canonical "here's why naive MAP fails and what actually works" result — worth stating as the takeaway.

• attribution to state plainly: **Fergus et al. 2006** — "Removing Camera Shake from a Single Photograph" — put blind deblurring on the map for photographs using a natural-image gradient prior + variational inference; **Levin et al. 2007/2009** clarified the *role of the prior* and *why the estimator design matters*.

• forward-ref: the learned successors (CNN/diffusion deblurring) in [[Deep learning]] — same prior, now learned from data instead of hand-modeled.

• the assumption we've been making: a **single** kernel convolved over the whole image (**shift-invariant** blur). Real **camera shake** violates it.

• why shake is *not* a convolution: hand shake is mostly **rotation** of the camera, not translation. Rotation moves different parts of the frame by **different amounts** (and in different directions) — corners smear more than the center, and the smear *curves*. There is **no single PSF** that, convolved everywhere, reproduces it. [fig-camera-shake-nonuniform]

• the better model (Whyte 2011/2014): the blurry image is a **weighted sum of the sharp image under many camera poses** — integrate over the camera's **rotational trajectory** during the exposure. Equivalently a **spatially-varying PSF** that is consistent because it comes from *one* underlying 3-D camera motion (far fewer unknowns than an independent kernel per pixel).

• consequence: deblurring becomes inversion of this **global, non-stationary** operator — still linear in $X$, still the data-fit + sparse-prior recipe, but the forward operator is the pose-integral, not a convolution (so **no single-FFT trick**; back to the iterative solvers of the previous chapter).

• also note (Whyte 2014): real shots have **saturated highlights**, which break the linear model — handling clipping is part of "realistic" deblurring.

• forward-ref (the optimistic flip side): instead of *fighting* an accidental, hard-to-invert blur, **engineer** the PSF so it's easy to invert and even encodes depth — **coded aperture / motion-invariant capture / wavefront coding** (Levin 2007; Dowski & Cathey 1995) in the coded-imaging / PSF-engineering part. "Make the forward operator nice" beats "invert a nasty one."

• **placement note**: color2gray and the colorization framing below are *not* deblurring, but they are the same **data-fit + prior** inverse-problem recipe this chapter teaches — kept here as the general "advanced inverse problem" instances (their *solvers* live in [[Poisson image editing]] / [[Edge-preserving techniques]]).

• the problem: convert color → grayscale **preserving contrast that luminance alone destroys.** Two colors can be **isoluminant** (different hues, *same* luminance) — naive "take the luminance / desaturate" maps them to the **identical gray**, erasing a contrast a viewer clearly saw. [fig-color2gray]

• the framing (Gooch et al. 2005): pose it as an **optimization** — find the grayscale image $g$ whose **gradients best match the perceived color differences** between neighboring pixels. Where colors differ (even isoluminantly), force a gray-level difference; carry the **sign** so ordering is consistent.

• objective (the gradient-domain shape, ties back to [[Poisson image editing]]): $\min_g \sum \lVert \nabla g - \theta\rVert^2$, where $\theta$ is a target gradient derived from **CIELAB** luminance + chrominance differences (a signed color-contrast). Solve the resulting **Poisson-type linear system** — *the same machinery as gradient-domain editing.*

• why it sits here: it is a **least-squares reconstruction from a target gradient field** — the inverse-problem / gradient-domain pattern again, with the "prior" being *match perceived contrast*. Short section; mainly a demonstration that the framework generalizes beyond restoration.

• encoding note: contrasts are computed in a **perceptually uniform** space (CIELAB), not linear RGB — state the space explicitly (L1/L2: work where differences mean what you want).

• decision (resolve "here or in edge-preserving?"): **introduce the framing here; develop the solver in [[Edge-preserving techniques]]** (*Edge-preserving optimization (colorization)*). Here it's *another instance of the chapter's recipe*; there it's the *worked edge-aware affinity solve.* Cross-ref both ways.

• the problem: a **grayscale photo plus a few color scribbles** → propagate color over the whole image, **stopping at edges** (don't bleed color across object boundaries). [fig-colorization-scribbles]

• the inverse-problem framing (Levin, Lischinski & Weiss 2004): the **unknown** is the per-pixel chrominance $U$; the **constraints** are the scribbles; the **prior** is *neighboring pixels with similar intensity should get similar color* — an **affinity** between pixels keyed on intensity difference. Minimize color variation subject to the scribbles: $\min_U \sum_x \big(U(x) - \sum_{x'\in N(x)} w_{xx'}\,U(x')\big)^2$, with weights $w_{xx'}$ large when intensities match. A **sparse linear least-squares** solve (the same CG/multigrid machinery as [[Efficient solvers]] and Poisson editing).

• 💡 **Big lesson (callback):** the weight $w_{xx'}$ = "how much these two pixels belong together, judged by their intensity difference" *is* an **affinity** — exactly **edge-preserving = affinity** (→ see Big lesson L4, first placed in the bilateral filter; the colorization solver is the optimization-form of that idea). This is the hinge to [[Edge-preserving techniques]], where the affinity/edge-aware machinery is developed in full.

• forward-ref: the **learned** alternative — predict plausible colors from a gray image with a CNN, no scribbles (Zhang, Isola & Efros 2016) — in [[Deep learning]] / colorization.

• the shape: an under-determined image-formation model + a statistical prior that makes it solvable — the **same thesis** as blind deblurring, even though the physics (atmospheric scattering, not blur) is different.

• the (physical) forward model: $Y(x) = t(x)\,X(x) + A\,(1-t(x))$ — each pixel is the clean radiance $X$ attenuated by **transmission** $t(x)$ (how much light survives the haze; $t = e^{-\beta d}$, $d$ = scene depth), **blended with airlight** $A$ (the bright, scattered atmospheric light). Haze = a depth-dependent fade toward $A$.

• why it's under-determined: per pixel we know $Y$ (3 numbers) but want $X$ (3), $t$ (1), and the global $A$ — more unknowns than measurements. Need a prior. [fig-dark-channel]

• the **dark-channel prior** (He, Sun & Tang 2009): in a haze-free outdoor patch, *some* color channel is **nearly zero** somewhere (shadows, colorful/dark surfaces) — the patch's $\min_c\min_{\text{window}} X_c \approx 0$. Haze *lifts* this dark channel (adds airlight everywhere), so **how bright the dark channel is ≈ how much haze ≈ $1-t$.** That single observation pins down the transmission map.

• the pipeline (state as steps): estimate $A$ from the haziest (brightest-dark-channel) pixels → estimate a coarse $t(x)$ from the dark channel → **refine the transmission's edges** so it follows real object boundaries (a matting / guided-filter / edge-aware solve — He's original used soft matting, later the **guided filter**, [[Edge-preserving techniques]]) → invert the model for $X$. [fig-dark-channel]

• the takeaway to land: *the prior is doing all the work.* The model is unsolvable as stated; one well-chosen statistical regularity (dark channel) turns it into a clean recovery — the **same moral as Wiener and blind deblurring**, just a different prior.

• xref: the refinement step ties into [[Edge-preserving techniques]] (guided filtering); the learned successors live in [[Deep learning]].

• **before the dark channel — physics from scattering.** **Cozman & Krotkov** ([@cozman-krotkov-1997]) turned the haze model around to *recover depth from scattering* — given the atmosphere, the depth-dependent fade *is* a depth cue (haze carries information, it doesn't only destroy it). **Narasimhan & Nayar**'s vision-and-weather work ([@narasimhan-nayar-2002]) laid down the systematic physics of imaging through fog, haze, and rain and how to invert it (classically from two images or a known airlight) — the physical foundation the single-image priors later shortcut.

• **learned dehazing.** Replace the hand-built prior with a network. **Zhang, Sindagi & Patel** ([@zhang-sindagi-patel-2019]) train a single network that **jointly estimates the transmission map and produces the dehazed image** end-to-end, instead of estimating $t$ by a prior and inverting separately — the learned successor to the dark-channel pipeline (forward-ref [[Deep learning]]). (Also flagged: **SDDE** — *queue exact ref / author to confirm*.)

• **out of the air, into the water.** **Sea-thru** ([@akkaynak-treibitz-2019]) corrects **underwater** images with the *right* physics: unlike atmospheric haze, water attenuates each wavelength differently and the veiling light varies with range, so a single global airlight is simply wrong — Sea-thru estimates wideband, range-dependent attenuation (from a known depth/range map) and removes the water's color cast where naive dehazing fails.

• **haze as a task, not just a nuisance.** **Sakaridis, Dai & Van Gool** ([@sakaridis-etal-2018]) synthesize fog onto clear images to build training data for **semantic scene understanding in fog**, so dehazing becomes a means to robust *perception* (driving, robotics) — closing back to the depth-from-scattering view that fog *carries* scene information.

• the meta-point that closes the inverse-problem run: every method across these two chapters is *an optimization with a prior* — but who **optimizes the pipeline itself**? When a restoration/ISP pipeline has knobs (filter weights, kernel parameters, the whole sequence of stages), you want to **tune them by gradient descent against a quality loss** — which means the pipeline must be **differentiable** and **fast**.

• **Halide** (Ragan-Kelley et al. 2013): a language that **separates *what* an image pipeline computes (the algorithm) from *how* it runs (the schedule)** — tiling, vectorization, parallelism, fusion. One algorithm, many schedules → portable high performance without rewriting the math. This is *the* tool the rest of computational photography is implemented in.

• **differentiable Halide** (gradient Halide, Li et al. 2018): add **automatic differentiation** through the pipeline → you can **backprop a loss to the pipeline's parameters** (or use the pipeline as a layer inside a network). Now "tune the deblurring/dehazing/ISP parameters" or "learn the filter" is just gradient descent through a fast, scheduled pipeline.

• why it closes the chapter: it's the **engine** under the whole "data-fit + prior, minimized" program — and the bridge to [[Machine learning]] (differentiable pipelines = the boundary where classical optimization meets learned components; cf. FlexISP, Heide 2014, as a differentiable-ISP example).

• scope note: keep this **conceptual** (separation of algorithm/schedule; differentiability as an enabler), not a Halide tutorial — point to the language for hands-on use.

• **Image Analogies** (Hertzmann et al. 2001): give an example pair **A → A′** (an image and its filtered/stylized version) and a new target **B**; synthesize **B′** so that *"A : A′ :: B : B′"* — learned **by patch matching** (for each B patch, find the best-matching A patch and copy its A′ counterpart). A single example teaches a filter/style — texture-synthesis machinery turned into a transfer, the **pre-deep ancestor** of neural style (cross-ref [[Deep learning]] and [[#Inpainting, texture synthesis]]).

• **color (palette) transfer** (Reinhard et al. 2001): the leanest classical method — make a target photo take on a reference's **palette/mood** by matching just the **mean and standard deviation** of each color channel. Convert both images to a **decorrelated color space** (l$\alpha\beta$ / Ruderman — axes statistically near-independent for natural scenes, so channels can be retargeted separately without cross-talk), then per channel **shift** to the reference's mean and **scale** by the ratio of standard deviations. Only **first- and second-order statistics** — no patches, no histograms — yet enough to swap a sunset's warmth onto a daylight shot. The classical, **pre-deep ancestor of neural style / color grading**: where Gatys later matches *deep-feature* second-order statistics (Gram matrices), Reinhard matches *raw-pixel* low-order statistics in a smart color space. (Forward-ref the learned / style-based color-grading successors → [[Deep learning]] and *Style transfer as image-to-image translation* below; the decorrelated transfer space ties to [[Human (and animal) vision and color]].)

• **two-scale tone / style transfer** (Bae, Paris & Durand 2006): transfer the **pictorial look** of a model photograph onto a target by **separating scales** — an edge-preserving **base / detail** split — and transferring the **large-scale tonal distribution** (a histogram-match of the base) and the **detail/texture** amount independently. Classic example: give a snapshot the tonal drama of an Ansel Adams print. Ties to the base–detail decomposition of [[Edge-preserving techniques]] and to histogram matching (BASIC).

• the classical takeaway: style here is **low-order statistics + patches** — global mean/variance (Reinhard), histograms, local texture, exemplar patches — matched by hand-designed pipelines. The arc rises in statistical order: Reinhard's first/second moments → histograms/patches → the deep-feature statistics below, where the learned methods replace "hand-designed statistic" with "deep-feature statistic."

• **neural style** (Gatys, Ecker & Bethge 2016): the breakthrough that **content = deep CNN features** (what is where) and **style = the *correlations* among those features**, captured by the **Gram matrix** $G_\ell=\phi_\ell\phi_\ell^\top$ at each layer. **Optimize an image** to simultaneously match the target's content features and the reference's style Gram matrices: $\min_{\hat I}\ \alpha\lVert\phi(\hat I)-\phi(I_c)\rVert^2 + \beta\sum_\ell\lVert G_\ell(\hat I)-G_\ell(I_s)\rVert^2$. Slow (an optimization per image), but it reframed style as *deep-feature statistics*. [fig-neural-style-content-style]

• **feed-forward / perceptual-loss style transfer** (Johnson, Alahi & Fei-Fei 2016): **train a network once** to produce the stylized output in a **single forward pass**, by minimizing the *same* content+style **perceptual loss** (deep-feature distance) Gatys optimized iteratively. Real-time stylization; the same **perceptual / feature loss** reappears as a training loss for super-res and as LPIPS (cross-ref [[Deep learning]]).

• **Markovian / patch-based neural style** (Li & Wand 2016): combine **MRFs with CNNs** — match style on **local neighbourhoods of deep features** (a patch/Markovian style term) rather than global Gram statistics, better preserving local structure. Sits **between Image Analogies (patches, no net) and Gatys (global feature statistics)** — patches, but in deep-feature space.

• **the GAN view**: when "style" is a whole **domain** (photos↔paintings, day↔night, summer↔winter), pose stylization as **learned image-to-image translation** — a generator maps one domain's look onto another, a discriminator keeps it plausible.

• **Pix2Pix** (Isola et al. 2017): **paired** translation — when you have aligned (input, styled-output) pairs, learn the map directly with a conditional GAN. The generic "learn the map between two image domains" tool.

• **CycleGAN** (Zhu et al. 2017): the **unpaired** version — when no aligned pairs exist (you have *photos* and *Monets*, but not the same scene as both), **cycle-consistency** ($B\to A\to B$ should return $B$) lets you learn the style mapping without pairs. The workhorse for "make my photo look like X" when X is only a *collection*.

• placement: these are surveyed as *models* in [[Deep learning]] (generative image-to-image); here they're the **learned, domain-level** end of the style-transfer spectrum. Forward-ref: text-/exemplar-driven stylization with diffusion ([[Generative AI and diffusion]]) is the current frontier.

• refs: Reinhard et al. 2001 (color transfer between images); Hertzmann et al. 2001; Bae, Paris & Durand 2006; Gatys et al. 2016; Johnson, Alahi & Fei-Fei 2016; Li & Wand 2016; Isola et al. 2017 (Pix2Pix); Zhu et al. 2017 (CycleGAN).

• **the problem**: given an image with a **masked region** (a scratch, a date-stamp, a removed power line, a missing JPEG block), produce a complete image where the hole is **plausible and seamless**. There is **no data in the hole** — the entire fill is the prior's invention. [`fig-inpaint-prior-spectrum`]

• **the organizing axis — weak → strong prior** (this is the chapter):

• **smoothness / PDE** (weakest): *"continue the surrounding color/edges smoothly."* Great for **thin** gaps; **blurs** anything large because smoothness has no notion of *texture*.

• **self-similarity / exemplar** (the workhorse): *"this image repeats itself — copy patches from elsewhere in the same image."* Texture synthesis, clone/heal, Criminisi object removal. Copies **real** structure (reconstruction-leaning), but only structure the image already contains.

• **database** (Hays & Efros): *"the answer is in some other photo"* — retrieve a matching scene from millions and graft a region in. The prior is *the rest of the internet*.

• **learned generative** (strongest): a net that has internalized $p(\text{image})$ **generates** a completion (context encoders → gated conv → diffusion inpainting). Most flexible, most prone to **hallucination**.

• **intuition-first framing**: not "Photoshop magic" — *"the algorithm answers a question it has no data for, by assuming the missing region looks like (a) its neighbours, (b) some other part of this image, (c) some other image, or (d) a typical image."* Those four assumptions are the four families.

• 💡 **the throughline**: this is L10 with the dial at max — see the box. Everything below is *which* prior.

• **the idea** (Bertalmío et al. 2000): mimic what a **museum restorer** does to a cracked painting — **continue the lines** arriving at the hole's boundary smoothly into it. Formally, **propagate the image's smoothness (its Laplacian) *along* the level-lines (isophotes)** so edges arriving at the hole are extended across it rather than blurred over. [`fig-pde-isophote-diffusion`]

• **why "along isophotes" matters**: naive diffusion (solve Laplace's equation in the hole — a membrane / harmonic fill) smears *across* edges → a soft blur. Steering the diffusion **along** the level-lines keeps an edge continuing as an edge. (Telea 2004 is a fast **marching-method** approximation.)

• **good for / hard limit**: **thin** structures — scratches, text overlays, fine cracks, scanner dust. For a **large** hole there is nothing to propagate but flatness → the fill goes **blurry and textureless** — the motivation for the exemplar/texture methods next.

• **placement note**: this is the **gradient-domain / Poisson** family in disguise — cross-ref [[Poisson image editing]] (💡 L9): seamless *cloning* edits the gradient field too; PDE inpainting edits it where there's *no source at all*.

• **the reframing**: a large hole in a *textured* region (grass, gravel, a sweater) can't be diffused — but it **can be synthesized**, because texture is **self-similar**: patches recur.

#### Non-parametric texture synthesis (Efros–Leung)

• **non-parametric texture synthesis** (Efros & Leung 1999): synthesize **one pixel at a time** — for the next unfilled pixel $p$, take its **already-known neighbourhood** $N(p)$, search the source for **windows that match it** (Gaussian-weighted SSD over the known part), pick a best match, and **copy its centre pixel**. No texture *model* is fit — the sample image **is** the model (a Markov-random-field assumption). [`fig-efros-leung-growth`]

• **the one knob — window size**: too **small** → loses large-scale structure (noise); too **large** → copies the source **verbatim** and is slow. Must straddle the texture's characteristic scale. Honest cost: **slow** (a full search per pixel) → the motivation for PatchMatch ([[#Patch match]]).

• **pseudocode block** (in the draft): the frontier loop — `while unfilled: pick the most-surrounded pixel p; SSD-match N(p) against source windows; copy a near-best match's centre into p`.

#### Image quilting (Efros–Freeman)

• **image quilting** (Efros & Freeman 2001): synthesize a **whole patch at a time** — lay down overlapping blocks copied from the source, and where two overlap, **cut along the minimum-error boundary** through the overlap (a tiny dynamic-program / 1-D graph cut) so the seam is invisible. Faster than pixel-at-a-time, and the seam-cut prefigures **GraphCut textures** (Kwatra 2003) and 💡 **L12**. [`fig-quilting-seam`]

• **parametric texture** (the road not taken): **Heeger & Bergen 1995** / **Portilla & Simoncelli 2000** match **statistics** of **steerable-pyramid** subband responses — the direct ancestor of **neural style's Gram-matrix** idea (cross-ref [[Style transfer]], [[Linear pyramids and wavelets]]).

• 💡 **L12 connection**: the patch + min-error-seam view is **discrete optimization on a graph** (match cost + seam cost) — the energy-is-the-design lesson (→ Big lesson **L12**).

• **clone brush**: the manual exemplar prior — **copy pixels from a chosen source region**. Pure copy → **visible seams** where lighting differs. The **healing brush** fixes that by transferring **texture/detail** but **matching the destination's color/illumination** — paste the source's *gradients* and re-level (the **gradient-domain / Poisson** trick — cross-ref [[Poisson image editing]], 💡 L9).

• **exemplar-based inpainting / object removal** (Criminisi, Pérez & Toyama 2004): automate the clone brush with texture-synthesis-style **patch copying** plus a crucial **fill order** — a **priority** $P(p)=C(p)\,D(p)$ (a **confidence** term × a **data/edge** term) fills **high-priority patches first**, so strong linear structures (a wall edge, a horizon) continue across the hole *before* flat regions. [`fig-object-removal`]

• **scaling → next sections**: the search is slow and limited to *this* image; **PatchMatch** ([[#Patch match]]) makes it interactive; a **database** (Hays & Efros) or a **learned net** lifts the "only this image" limit.

• **the provocation**: if exemplar methods are limited to *this image's* content, **use a different image.** Hays & Efros 2007 fill a hole by **retrieving, from millions of photos, scenes that match the hole's surround**, then **seamlessly graft** a region from a good match (alignment + a graph-cut seam + Poisson blend). [`fig-inpaint-prior-spectrum`, database panel]

• **the lesson — "the internet is your prior"**: with a big enough database, *some* photo already contains a plausible completion; you **retrieve** rather than model. The non-parametric prior taken to web scale.

• **honest caveats**: it grafts a **semantically different** scene — coherent and photoreal, but not *this* scene's truth (a vivid L10 "plausible ≠ measured"); needs a huge database and good scene matching.

• **why it's the hinge to deep learning**: "retrieve the right completion from millions of images" is what a **generative net** does *implicitly* — it has compressed those millions into weights and can **synthesize** the completion.

• **the move** (💡 L8, [[Deep learning]]): replace the hand-built prior with a **learned** one — a net trained to **regenerate masked regions**.

• **context encoders** (Pathak et al. 2016): the first CNN inpainter (encoder–decoder, reconstruction + adversarial loss) — captures **semantics** but early versions were blurry.

• **partial / gated convolutions** (Liu et al. 2018; Yu et al. 2019): fix the core nuisance — a vanilla conv near a hole mixes in **garbage masked pixels**. Partial conv renormalizes by the *valid* fraction; gated conv *learns* a soft mask gate → robust to **free-form** holes, the practical "remove this object" workhorse.

• **diffusion inpainting** (Lugmayr et al. 2022, RePaint): the strongest prior — treat the hole as a **measurement constraint** and, at every denoising step, **pin the known region** to the (renoised) real pixels: $x_{t-1}\!\leftarrow\!\text{mask}\odot q(x^{\text{known}}_{t-1}) + (1-\text{mask})\odot \mathcal{D}_\theta(x_t)$. This is **exactly PnP/RED posterior sampling** (💡 L11) with operator = "keep the known pixels" — the continuation of [[#Super-resolution and image priors]].

• **forward reference**: the architectures live in [[Deep learning]] / [[Generative AI and diffusion]]; here they're the **strong-prior end**. Honest note: most flexible **and** most prone to **hallucinate** — an inpainted region is **not evidence** (L10 ethics).

• **the framing — clipping is a hole**: a **blown highlight** is a region where the sensor **saturated** — the true radiance above the clip is *unmeasured*. So **highlight recovery is inpainting a clipped region**: fill the saturated pixels using the **unclipped channels** and surrounding texture as the prior. [`fig-highlight-recovery`]

• **two angles**: (a) **specular vs diffuse separation** — recover the diffuse layer under the specular add-on (cross-ref [[#Illumination related effects in a single image]]); (b) **per-channel clipping recovery** — when only one channel clipped, the others carry the hue, so extrapolate the missing channel (a small, well-posed inpainting).

• **why it lives here**: same L10 idea — *recover what the measurement destroyed* (by clipping rather than masking); the single-image cousin of **HDR merge** (capture the highlight with a shorter exposure instead of guessing it — forward-ref [[Multiple exposure imaging]]).

• **the idea** (Jojic, Frey & Kannan 2003): summarize an image (or video) by a **small "epitome"** — a miniature holding the **patch statistics** of the whole, learned so that every original patch has a close match somewhere inside it. A **generative patch model**: the big image is explained as a quilt of epitome patches.

• **uses**: denoising, inpainting, super-resolution, and texture transfer all fall out of "find each patch's place in the epitome, read back a clean version" — the **patch-NN** idea of [[#Patch match]] with a *learned compact dictionary* instead of the image itself.

• **place**: the pre-deep cousin of today's learned patch / latent priors (**L8 / L11**); a bridge from exemplar texture synthesis to modern generative models.

• **the shared sub-problem**: inpainting, texture synthesis, retargeting, NL-means denoising, Image Analogies — all need, for each output patch, **the most similar source patch**. Collect those answers → a **nearest-neighbour field (NNF)**: an **offset** $f(p)=\Delta$ per pixel s.t. $P_{\text{in}}(p+\Delta)$ best matches $P_{\text{out}}(p)$. [`fig-nnf-field`]

• **the cost wall**: computed exhaustively, each patch is compared to **every** candidate → quadratic; kd-trees help but stay far from interactive on megapixels. *This* is why Efros–Leung and Criminisi were slow, and why "Content-Aware Fill" didn't exist as a click-and-wait tool until the NNF could be computed in real time.

• **the reframing**: compute the NNF **approximately, but fast** — a **randomized heuristic** (PatchMatch) or a **global energy minimization** (Shift-Map). The rest of the chapter is those two.

• **the two facts it exploits**:

1. **coherence** — adjacent output patches usually map to **adjacent** source patches, so **one good offset is good for its neighbours.**

2. **the birthday-paradox / law of large numbers** — over a whole image **some** patches get a good match by **random guessing**; you just need to **spread** the lucky hits.

• **the algorithm — three moves** [`fig-patchmatch-three-steps`]:

1. **random initialization**: every pixel gets a **random** offset. Mostly bad — a few good.

2. **propagation**: scan (alternating raster order each iteration); for $p$, also **try the offsets its scanned neighbours used** (shifted by one) → good offsets **flood-fill** across coherent regions.

3. **random search**: refine the best by testing random offsets in an **exponentially shrinking window** ($w\cdot\alpha^i$) — large jumps escape local optima, small jumps fine-tune.

• iterate 2–5 times → a clean NNF in **milliseconds**, orders of magnitude faster than exhaustive search.

• **pseudocode block** (in the draft): the init-then-sweep loop — `init f randomly; repeat passes (alternating scan): for each p: propagate from neighbours, then random-search in a shrinking window; return f`.

• **why it converges fast**: propagation makes a single lucky match **infect** an entire coherent region in one scan; random search avoids getting stuck. *Randomness + coherence beats brute force.*

• **generalized PatchMatch** (Barnes et al. 2010): k nearest neighbours + **rotations/scales** — needed for matching transformed content (style transfer, denoising).

• 💡 **why it sits next to Shift-Map**: PatchMatch answers "where does each patch come from?" **heuristically and fast**; Shift-Map answers the **same** question **optimally and slower** (next).

• **interactive hole filling / Content-Aware Fill**: run PatchMatch between hole and rest, copy best-matching patches in, iterate coarse-to-fine for global coherence — the engine that turned exemplar inpainting into a **real-time** tool. The *objective* it approximately optimizes is **bidirectional similarity** (Simakov et al. 2008 — output should be *coherent* and *complete*). [`fig-patchmatch-apps`, hole panel]

• **image retargeting** (content-aware resize): change aspect ratio while **preserving important content** (vs **seam carving**, Avidan & Shamir 2007, the greedy cousin) — reshape without squashing faces/buildings. [`fig-patchmatch-apps`, retarget panel]

• **reshuffle**: **drag an object** to a new location; re-synthesize the rest so the scene stays coherent. [`fig-patchmatch-apps`, reshuffle panel]

• **and beyond**: **video completion** (Wexler et al. 2007), **Image Melding** (Darabi et al. 2012 — PatchMatch + gradient-domain), denoising self-similarity, the search inside Image Analogies (cross-ref [[Style transfer]]).

• **honest framing (L10 callback)**: all still **copy** real content (reconstruction-leaning within the image); the *generative* successors (diffusion inpainting/outpainting) **invent** content — more powerful, less faithful.

• **the reframing**: instead of a heuristic search, decide **per output pixel** a single **shift** $M(p)$ into the input by **minimizing a global energy**, then read the output by copying `input(p + M(p))` — the output is a **rearrangement** of input pixels under a smooth offset field. [`fig-shiftmap-labeling`]

• **the energy** (💡 L12): $E(M)=\sum_p E_{\text{data}}(M(p)) + \lambda\sum_{(p,q)} E_{\text{smooth}}(M(p),M(q))$.

• **data term** encodes the task: retargeting (force output size / drop columns), object removal (forbid shifts into the removed region), reshuffle (pin content to its new place).

• **smoothness / seam term** penalizes **inconsistent neighbouring shifts** by **color + gradient discontinuity** — cheap seams fall on **real edges** (same principle as quilting's min-error cut).

• **the solver — graph cuts / α-expansion** (Boykov et al. 2001), often **coarse-to-fine** (the label set — all shifts — is large) → a **globally (near-)optimal** offset map.

• **PatchMatch vs Shift-Map — the duality**: both produce an **offset field** rearranging input into output. PatchMatch = randomized **heuristic**, fast, approximate, great for dense synthesis (fill, reshuffle). Shift-Map = a **global graph-cut energy**, slower, optimal for its energy, with clean explicit seams — great for retargeting / removal. This is the **L4→L12** progression: an **affinity** (patch similarity / seam cost) used by a **fast local rule** or a **global cut** — *the affinity is the design; the solver is a choice.*

• **cross refs**: seam carving (Avidan & Shamir 2007) is the greedy special case Shift-Map generalizes; the seamless blend after a cut is the **Poisson** solve ([[Poisson image editing]], L9); the learned successor is [[Generative AI and diffusion]] (L11).

• **the task**: produce a **binary mask** — these pixels are the subject, those are not. Three classic framings, the *same* lesson (**L12**) with three different optimizers — the energy/affinity you pick is the design, not the solver.

#### Interactive graph-cut segmentation

• **interactive graph-cut segmentation** (Boykov & Jolly 2001): treat the image as a **grid graph**; the user paints a few **FG** and **BG** strokes. Build a color model from the strokes (a **data/region term** $D_p$), add an **edge-aware smoothness term** $V_{pq}\propto e^{-\lVert C_p-C_q\rVert^2/2\sigma^2}$ (cheap to cut where colors differ), and minimize $E=\sum_p D_p+\lambda\sum_{pq}V_{pq}$ by **min-cut/max-flow**. For two labels this is **globally optimal** — *why* graph cut beats greedy region-growing. [`fig-segmentation-graphcut`]

#### GrabCut: less input, same cut

• **GrabCut** (Rother et al. 2004): the user drags a **loose rectangle**; fit **Gaussian-mixture color models** to FG/BG, run the cut, **re-estimate** the models, **iterate**. A rectangle plus a couple of strokes → a clean mask — the interaction model "select subject" tools descend from.

#### Two other framings: normalized cuts and intelligent scissors

• **normalized cuts** (Shi & Malik 2000): the **spectral** framing — cut a pixel-affinity graph into balanced pieces by a **normalized** cut cost, solved by an **eigenvector** of the graph Laplacian (a relaxation) → *unsupervised* grouping. Same affinity, different optimizer. [`fig-segmentation-three-framings`]

• **intelligent scissors / live-wire** (Mortensen & Barrett 1995): the **boundary** framing — a **least-cost path** (dynamic programming, costs low along strong edges) **snaps** to the contour as you trace. Energy on the *boundary*, optimizer = shortest-path. Three energies — region, spectral, boundary — one **L12** lesson. [`fig-segmentation-three-framings`]

#### The bridge to soft selection

• **the bridge to soft selection**: every method here returns a **hard 0/1** mask, which through **hair** looks scissor-cut — jagged, missing wisps, fringed. The fix is to relax the label to a **continuous $\alpha\in[0,1]$** — **matting**, next.

• **the model**: each observed pixel is a **mixture**, $C = \alpha F + (1-\alpha)B,\ \alpha\in[0,1]$ — $\alpha=1$ pure foreground, $\alpha=0$ pure background, and the *interesting* pixels are **in between** (a strand of hair, a motion-blurred edge, an out-of-focus border, frosted glass). The matte $\alpha$ is the **coverage / transparency** channel. [`fig-matting-equation`]

• **compositing = the forward direction** (the "over" operator, Porter & Duff 1984 — *queue ref*): given $F,\alpha$ and a new $B$, the composite is one **per-pixel lerp**. (In practice colors are **premultiplied**, $F'=\alpha F$, so over is $C=F'+(1-\alpha)B$ — cheaper, correct under filtering.) This part is easy.

• **matting = the inverse, brutally ill-posed**: from one photo you observe only $C$ (3 numbers) and must recover $F$ (3), $B$ (3), $\alpha$ (1) — **7 unknowns, 3 equations** per pixel.

• 💡 **callback to L10 (the prior is not optional)**: every method adds information — a user **trimap** + a **prior/affinity** (natural matting), a **known background** (chroma key), or an **extra measurement** (depth/IR/multi-flash). Same moral as super-resolution and dehazing. (→ see Big lesson **L10**, [[#Super-resolution and image priors]].)

• **the trimap** is the standard input: a three-way label — **definitely FG**, **definitely BG**, and a thin **unknown** band $\Omega$ straddling the boundary (the hair, the blur). From a scribble, a dilated mask, or learned.

• **why it's enough**: in the definite regions $\alpha$ is pinned and $F$ or $B$ is *observed*; the algorithm solves $\alpha,F,B$ only in the **narrow unknown band**, with nearby known colors as anchors — "7 unknowns everywhere" → "a few unknowns in a thin strip with strong boundary conditions." [`fig-trimap-to-alpha`]

• **the through-line**: *less user input is the research arc* — hard scribbles (graph cut) → loose rectangle (GrabCut) → trimap (matting) → **nothing** (learned/SAM).

• **Bayesian matting** (Chuang et al. 2001 — *queue ref*): a **per-pixel MAP** — model nearby FG/BG colors as Gaussians, find the $(F,B,\alpha)$ best explaining $C$, sweep outward from known regions. Intuitive but **local** (neighbouring alphas needn't agree).

• **closed-form matting** (Levin et al. 2008 — *queue ref*): **eliminate $F,B$** — assume in a small window FG and BG each lie on a **color line**, so $\alpha$ is locally an **affine function of color**, and solve $\hat\alpha=\arg\min_\alpha \alpha^\top L\,\alpha$ s.t. the trimap, where $L$ is the **matting Laplacian** (a sparse **pixel-affinity** matrix). A **sparse linear solve**. [`fig-trimap-to-alpha`]

• 💡 **the hinge — recurrence of L4**: the matting Laplacian $L$ **is an affinity matrix** — the *same object* as the bilateral weight, the graph-cut smoothness term, and the **colorization** solve ([[#Colorization as an optimization (the inverse-problem framing)]]). "**Propagate $\alpha$ along affinities, anchored by the trimap**" is colorization with $\alpha$ in place of chroma. (→ Big lesson **L4**, [[Bilateral filtering]]; the *cut* form is **L12**.)

• **robust / sampling hybrids** (Wang & Cohen 2007 — *queue ref*): combine color-sampling with the affinity smoothness — the workhorse just before the learned era.

• **the trick**: make the inverse *well-posed by construction* — shoot against a **known, uniform, saturated background** (green/blue screen) → $B$ is **known** (and far from skin/hair), so only $\alpha,F$ remain (Smith & Blinn 1996 — *queue ref*). [`fig-key-vs-measure`, left]

• **green vs blue**: **green** dominates today (two green photosites per Bayer cell → less noise; rarer in skin/clothing); **blue** was the film standard.

• **practice / failure modes**: **spill** (green bounce on edges → de-spill), **uneven screen lighting** (the "known" $B$ isn't constant), **fine detail** (hair still needs a soft $\alpha$ at the boundary). *Why studios obsess over even screen lighting* — it keeps $B$ genuinely known.

• **the moral**: chroma key removes unknowns by **controlling the scene**.

• **the other road**: **add a measurement** that distinguishes FG from BG — **computational illumination** (light/sense so the *capture* hands you the matte).

• **depth matting**: a **depth sensor** (ToF, stereo, dual-pixel, LiDAR) gives "near = foreground" independent of color — phone **portrait mode**, video-call background blur. The depth edge is coarse → used as a **trimap generator**, refined with closed-form/guided matting (same affinity solve; cf. the guided-filter transmission refinement in [[#Dehazing]]).

• **IR / "Disney" matting**: light the background in the **infrared** (a retro-reflective IR background; IR-keyed matting) → a *second IR image* is a built-in green screen with **no visible colored screen** to spill/relight.

• **flash / multi-flash matting** (*queue ref*): a **flash** falls off as $1/d^2$, brightening the *near* foreground far more than the far background → differencing flash/no-flash (or multi-flash occlusion shadows) separates FG/BG by **illumination falloff**. [`fig-key-vs-measure`, right]

• **the moral**: these spend a *sensor or a light* to make the matte well-posed, not *user input* or a *prior* — the computational-illumination thread.

• **rotoscoping**: the hand-craft ancestor — tracing a moving subject **frame by frame**; modern tools interpolate **keyframed splines/masks** — segmentation with a **temporal** axis.

• **the new requirement — temporal coherence**: a per-frame matte that **flickers** looks worse than a slightly-wrong but **stable** one, so video matting adds a **temporal smoothness / propagation** term (track, or optical-flow-warp the matte forward) — the graph/affinity now spans a video volume.

• **applications**: green-screen VFX, live virtual backgrounds, object removal/replacement, the **selection** layer under any video edit (ties to [[Non-photorealistic rendering]]). Learned video matting (Lin et al. 2021 — *queue ref*) does this with a recurrent net.

• 💡 **callback to L8**: deep methods keep the *same skeleton* — select, then recover $\alpha$ — but replace the hand-built color model / affinity / trimap-propagation with a function **fit to data**. The matting equation and "cut then blend" don't change; the prior does. (→ Big lesson **L8**, [[Deep learning]].)

• **SAM — Segment Anything** (Kirillov et al. 2023): a **promptable** model — click/box/"everything" → clean masks for **arbitrary objects**, no per-class training; collapses GrabCut's interaction to a prompt and its color model to a pretrained net. The "**cut anything**" front-end feeding masks/trimaps downstream.

• **deep alpha matting** (Xu et al. 2017 — *queue ref*; trimap-free MODNet / robust matting, Lin et al. 2021 — *queue ref*): a network regresses $\alpha$ (and $F$) directly, learning the hair/edge prior; trimap-free variants push user input to **zero** (the end of the "less input" arc), only as good as their training distribution.

• placement: taught as **models** in [[Deep learning]]; here the **learned-prior** end of the same select-then-recover spectrum. Generative *object insertion* (re-render the blended result incl. shadow) is [[Generative AI and diffusion]] (**L11**).

• **the problem after the cut**: even with a *perfect* $\alpha$, a raw paste looks "**stickered on**" — different light, white balance, contrast, **grain** than the new background. The matte was right; the *appearance* is wrong. [`fig-harmonization`]

• **two layers of fix, both pointing to EDGES MATTER**:

• **seamless blending** — **Poisson / gradient-domain** (Pérez et al. 2003) pastes the foreground's **gradients**, re-deriving the boundary color from the background (**L9**, [[Poisson image editing]]); **multi-band / pyramid blending** (Burt & Adelson 1983) blends each band over a band-appropriate width (developed in [[Linear pyramids and wavelets]]). *Used here, not re-derived.*

• **global appearance harmonization** — match the foreground's **color distribution, illumination direction, contrast and grain** (classical: transfer color statistics; learned: a net predicting a harmonized composite). Fixes lighting *consistency*, which Poisson alone doesn't (Poisson hides the *seam*, not a wrong *shading direction*).

• **the takeaway**: a believable composite needs **all three** — a correct **matte** (the cut), a seamless **blend** (Poisson/pyramid), and consistent **appearance** (harmonization).

• **resolving the stub's placeholders** — these belong to other parts (*cut here, blend there*):

• **Poisson / gradient-domain blending** → [[Poisson image editing]] (EDGES MATTER); used above (**L9**).

• **pyramid / multi-band blending** → [[Linear pyramids and wavelets]] (Burt & Adelson 1983).

• **interactive digital photomontage** (Agarwala et al. 2004) → the canonical *cut-then-blend* system: a **graph-cut seam** picks which source photo per pixel (group photo with everyone smiling, clean-plate removal, extended DoF), then a **Poisson blend** hides seams — exactly this part's two halves wired together (graph cut = **L12**; Poisson = **L9**); placed at the seam/blending crossover in EDGES MATTER, showcased here.

• **multi-image fusion** (HDR merge, exposure fusion, focus stacking, flash/no-flash) → [[Multiple exposure imaging]] (and the [[#Recap: tone mapping]] recap); shares "weighted per-pixel combine in a pyramid" with pyramid blending but combines **exposures/focus**, not **objects**.

• the one-line map: **segmentation/matting = the cut (this part); Poisson/pyramid/photomontage = the blend (EDGES MATTER); HDR/fusion = the multi-capture combine ([[Multiple exposure imaging]]).**

• **the decomposition**: split one image into two **intrinsic layers** — **reflectance** $R$ (the surface's albedo, independent of lighting) and **shading** $S$ (the illumination, independent of surface). Per channel $I = R\cdot S$; in **log** it's additive, $\log I = \log R + \log S$ (**L2** — why illumination work lives in log/linear, not gamma). [`fig-intrinsic-decomp`]

• **why it's the umbrella**: once you can name "this variation is *light*, that is *surface*," every effect here is an edit on one layer — **white balance** corrects the *color* of $S$; **shadow removal** flattens a dark region of $S$; **highlight removal** strips a specular add-on; **reflection removal** un-sums two whole $I$'s.

• **why it's hard**: infinitely many $(R,S)$ multiply to the same $I$ — the multiplicative ill-posedness in the box. You need a prior on *which* variations are reflectance and which are shading.

• **Retinex** (Land & McCann 1971) — the **classic gradient prior**: assume **reflectance changes are sharp** (a painted edge) while **shading changes are smooth** (light falls off gradually). Threshold the (log-)gradient: **big jumps → reflectance**, **gentle ramps → shading**; then **Poisson-integrate** each (**L9**). Crude but captures the whole intuition: *edges are surface, ramps are light.* [`fig-retinex-thresholding`]

• **SIRFS** (Barron & Malik 2015 — *queue ref*): jointly recover **Shape, Illumination, Reflectance** by **MAP** with hand-built priors (reflectance piecewise-constant/sparse, shape smooth, illumination low-frequency). Same data-fit + prior recipe; learned successors (CNNs predicting $R,S$) in [[Deep learning]] (**L8**).

• **the takeaway**: intrinsic decomposition is *the* generalization; the rest is "I care about one layer, and I have a cue that pins it down."

• **the assumption that breaks**: ordinary white balance estimates **one** illuminant for the whole frame and **divides** it out (**L1**, [[Color technology]]) — right only under a *single* color of light.

• **the failure**: real scenes mix lights — **warm tungsten** vs **cool daylight** through a window; flash + ambient; sun + open-shade blue. One global correction neutralizes *one* region and casts the other: fix the tungsten side and the window goes blue. [`fig-multi-illuminant-wb`]

• **the fix — per-pixel light mixture** (Hsu et al. 2008): model each pixel as a **blend of two illuminant colors**, $I(x) = (\alpha(x)L_1 + (1-\alpha(x))L_2)\cdot R(x)$, solve for the **mixing map** $\alpha(x)$ — a **spatially-varying** white balance — then divide out the *local* light. The two colors come from the user, clipped highlights, or clustering.

• **why it's intrinsic-image**: recovering the *color* of the shading layer as it varies — a chrominance-only intrinsic split; the map $\alpha(x)$ is **smooth** (light varies gently) — the Retinex prior on illuminant *color*.

• **encoding note**: divide in **linear light** (mixing/dividing illuminants is linear in radiance — **L2**).

• **cross-ref**: the *easy* multi-image version (flash/no-flash, the flash's known color disambiguates) is in **Computational illumination (Advanced)**; learned single-image in [[Deep learning]].

• **the problem**: shoot through a window → **two images summed**, the **transmitted** scene ($T$) plus a **reflection** ($R$): $I = T + R$ (this one is **additive**, unlike $R\cdot S$; **L2**). Pulling them apart from one $I$ is hopelessly under-determined — pure prior territory (**L10**).

• **the cue zoo** (each a different prior breaking the tie): [`fig-reflection-removal-cues`]

• **defocus / blur**: you focus on the *transmitted* scene → the **reflection is out of focus** (weak, broad gradients). Assign sharp gradients to $T$, soft to $R$ (Fattal 2008 — a Retinex-like split on *focus*).

• **ghosting**: a *thick* pane reflects twice → the reflection appears **doubled/shifted**; that known double-impulse lets you identify and subtract $R$.

• **sparse-gradient prior** (Levin & Weiss 2007 — *queue ref*): both layers have **sparse, heavy-tailed gradients**; the sum has *too many* edges → find the $(T,R)$ split **minimizing total edges** (optionally with user scribbles). Same sparse-gradient prior as blind deblurring.

• **polarization**: light off glass is **partially polarized** (Brewster) → a polarizer attenuates $R$ differently from $T$. *(The genuine multi-image method — rotate a polarizer, two frames — lives in **Computational illumination (Advanced)**; single-image methods try to get this without the extra shot.)*

• **learned** (Zhang et al. 2018 — *queue ref*): a CNN trained on synthesized $T+R$ pairs predicts $T$ with a **perceptual loss** so the layer *looks* like a real photo (**L8**).

• **the takeaway**: "$I = T + R$ has no answer; pick the cue (focus, ghosting, edge-sparsity, polarization, a learned prior)." A cousin of intrinsic images — un-summing two illuminations.

• **what a shadow *is*, intrinsically**: a region where the **shading** $S$ drops (less light) while **reflectance** $R$ is **unchanged** — same paint, just darker. So shadow removal is an **edit on the shading layer only**: flatten the darkening, keep the texture. This is why it lives under intrinsic images.

• **the detection cue (the crux)**: tell a **shadow edge** (shading discontinuity) from a **material edge** (reflectance): a shadow edge changes **brightness but not hue much** (same surface color), is often **soft** (penumbra) and geometrically smooth; shadow regions are lit by **skylight** (bluer) not direct sun (warmer) — a color tell.

• **removal as a gradient-domain edit** (**L9**): label the shadow *boundary* gradients, **zero them**, and **Poisson-reconstruct** — interior texture (small gradients) preserved, the big shadow step removed. Same machinery as gradient-domain compositing. Refine the matte with the **fast bilateral solver** (Barron 2017) so the correction follows the real penumbra (**L4**).

• **the penumbra subtlety**: a hard binary mask leaves a dark halo; you need a **soft shadow matte** and to **scale** (multiply the shadowed radiance back up by the recovered shading ratio — a multiplicative correction in linear light, **L1**).

• **cross-ref**: the easy multi-image version (**flash/no-flash**) is in **Computational illumination (Advanced)**; learned shadow removal in [[Deep learning]].

• **the problem**: glossy surfaces (skin, leaves, plastic, wet roads) show **specular highlights** — the light source reflected, carrying the *illuminant's* color, not the surface's; they blow out texture and fool color/material estimation. We want the **matte (diffuse-only)** image.

• **the dichromatic reflection model** (Shafer 1985 — *queue ref*): reflected light is the **sum of two terms** with **different colors** — a **diffuse/body** term in the **surface's** color + a **specular/interface** term in the **illuminant's** color: $I(x) = m_d(x)\Lambda_{\text{body}} + m_s(x)\Lambda_{\text{illum}}$. The falloffs vary per pixel, but there are only **two color directions**.

• **the geometric picture (why separable)**: plot a glossy patch's pixels in RGB → an **"L"/dog-leg**: a **line** along the body color (matte shading) with a **spur** toward the **illuminant white** (the specular ramp). Identify the two directions (Lee 1979; Klinker–Shafer–Kanade 1988 — *queue ref*), then **project each pixel back onto the diffuse line**. [`fig-dichromatic-model`]

• **"fake polarization"**: a **real** specular killer is a **polarizing filter** (specular reflection is polarized); doing the *same job* purely **computationally from one image** via the dichromatic model is the "fake polarization" — the polarizer's matte look **without** the filter or extra shot. (Lee 1979 also used the highlight's illuminant color for **color constancy** — a highlight is a free white reference.)

• **why it's an intrinsic-image cousin**: the specular term is an **additive contamination of the shading layer** in the *illuminant's* color; removing it recovers the diffuse $R\cdot S$. A **color-direction** prior rather than a gradient prior — different cue, same un-mixing goal.

• **cross-ref**: the genuine optical version (rotate a polarizer) and multi-light specular removal are in **Computational illumination (Advanced)**; learned separation in [[Deep learning]]. **Encoding**: separate in **linear light** so the dichromatic *sum* is physically additive (**L2**).

• **the goal**: a **depiction**, not a reproduction — painterly, sketch, or cartoon, **emphasizing structure** and **suppressing clutter**. Closer to *illustration* than photography; the value is in **what's left out**.

• **the one idea — structure-aware abstraction**: keep what the eye uses (strong **edges**, **large-scale tone**, the silhouette), simplify the rest (fine texture, mid-tone noise, busy backgrounds). **L13** (separate the subject, commit) executed in pixels, using **L4** (edge-preserving) machinery.

• **intuition-first**: a cartoon = **flat color regions + bold outlines.** Flat regions come from an **edge-preserving smoother** (don't blur across edges); outlines from an **edge detector**. That two-part recipe is the whole field in one sentence.

• **the idea** (Hertzmann 1998 — *queue ref*): repaint the photo with **discrete brush strokes**, **coarse-to-fine** — big rough strokes over a heavily-blurred version, then progressively **smaller** strokes **only where** the painting still differs from the (less-blurred) reference. Detail appears where needed; flat areas stay loose. [`fig-npr-painterly-layers`]

• **the structure that makes it look good (not random daubs)**:

• **orientation**: strokes run **perpendicular to the image gradient** (along the "grain") — following a cheek's contour or a sky's flow (**L9**);

• **size = scale**: coarse strokes on coarse pyramid levels, fine on fine — a direct use of the **Gaussian / image pyramid** ([[Linear pyramids and wavelets]]);

• **curved strokes**: trace along the orientation field for long painterly strokes, stopping at edges;

• **error-driven placement**: add a stroke only where canvas-vs-reference error exceeds a threshold → the painting **converges** to the photo at the chosen abstraction level.

• **the parameters are the *style***: stroke size range, opacity, jitter, curvature, blur radius → impressionist vs pointillist vs expressionist from *one* algorithm. (The **Gaussian pyramid** sets stroke scale and the blurred references.)

• **the course p-set**: implement the **layered, gradient-following brush painter** — build the pyramid of blurred references, place oriented strokes coarse-to-fine, explore a couple of brush/parameter styles on your own photo. [`fig-npr-brush-pset`] Use the photos-from-Fredo / sourced images for the worked example.

• **the canonical real-time pipeline** (Winnemöller et al. 2006 — *queue ref*): [`fig-npr-abstraction-pipeline`]

1. **flatten regions** with a **bilateral filter** (iterated) — smooths within objects, **stops at edges** (**L4**), collapsing texture into flat cartoon regions; optionally **quantize** color/luminance into a few bands (the posterized look);

2. **draw lines** with a **Difference-of-Gaussians**, $D = G_{\sigma_1}*I - G_{\sigma_2}*I$, thresholded into **black ink edges**;

3. **composite**: flat quantized color **+** DoG lines = a cartoon, in real time, even on video.

• **why this *is* EDGES MATTER**: the bilateral step is literally the [[Edge-preserving techniques]] base/detail filter (**L4**) — here used to **destroy** detail deliberately, not enhance it. Same tool, opposite intent.

• **flow-based DoG / coherent lines** (Kang et al. 2007 — *queue ref*): plain DoG gives **broken, noisy** edges; compute a smooth **edge-tangent flow** and run the DoG **along** it → **long, clean, coherent** lines like a confident pen. [`fig-npr-flow-dog`]

• **temporal coherence** (for video): filter with motion-awareness so flat regions and lines don't **flicker** frame-to-frame.

• **eye-tracked, perceptually-guided abstraction** (DeCarlo & Santella 2002, *Stylization and Abstraction of Photographs*, SIGGRAPH): the chapter's deepest answer to *what to keep*. Instead of inferring importance from edges alone, they **record where a viewer actually looks** (an eye-tracker) and build a **hierarchical, multi-scale** abstraction that **preserves detail at the fixated, meaningful regions and simplifies everything else** — bold flat color and clean bounding contours from a segmentation hierarchy, with detail *dialed in by measured visual attention*. NPR reframed as **directing the eye**: the abstraction reads as right because it is tied to **perception**, not just to gradients (**L13** — separate the subject and commit). This is the model of *meaningful* (importance-driven) abstraction, and the natural complement to the bilateral + DoG pipeline above, which abstracts the frame uniformly.

• **Image Analogies** (Hertzmann et al. 2001): give one example pair **A → A′** (a photo and its hand-stylized version) and a new photo **B**; synthesize **B′** so *A:A′ :: B:B′* by **patch matching** — *learn the style from one example*. The pre-deep ancestor of neural style; developed in [[#Style transfer]] (and surveyed in [[Deep learning]]) — appearing in both by design.

• **the bridge to neural style**: classical NPR hand-codes *what to keep* (edges, strokes, flat regions); **neural style** (Gatys 2016) replaces those rules with **deep-feature statistics** (content features + Gram-matrix style), and **feed-forward** stylization (Johnson 2016) makes it real-time — the same painterly goal, the prior now **learned** (**L8**). Full treatment in [[#Style transfer]] / [[Deep learning]].

• **the takeaway**: across painterly strokes, bilateral+DoG cartoons, and neural style, the constant is **abstraction guided by structure** — decide what carries meaning, keep and exaggerate it, simplify the rest. NPR is **L4 + L13** turned into a rendering style.

• **the idea**: instead of strokes or edges, **partition** the image into a few large regions and fill each with a **flat color** (its mean) — *abstraction by tessellation*. Pick the tiling to respect structure: a **Voronoi / centroidal** tessellation ("**stained glass by numbers**"), a **Delaunay low-poly** triangulation seeded at salient/edge points, or **superpixels** (SLIC). Same NPR thesis — *abstraction guided by structure* (**L4**) — with "what to keep" carried by **region boundaries** rather than brush orientation.

• **the dial = number / shape of regions**: few large cells → bold, poster-like; many small cells → near-photographic. Edge-aware seeding keeps tiles from straddling strong contours (the [[Bilateral filtering]] / segmentation link).

• **p-set lineage**: the "stained glass by numbers" make-your-own assignment; kin to **photomosaics** ([[10 Many images and photo collections#Photomosaics, Salavon]]) and to **graph-cut region labelling** ([[Seam optimization]]).

• **the idea**: reproduce a continuous-tone image with **discrete marks** — the classic print problem — but make the *marks themselves* expressive. Ordinary **halftoning** trades spatial resolution for tone (clustered-dot screens, **ordered dithering**, **error diffusion** à la Floyd–Steinberg); **artistic screening** (Ostromoukhov & Hersch) replaces the regular dot screen with **shaped, image-bearing screen elements** so the halftone microstructure carries a *second* picture, lettering, or texture. Tone is still right at a distance; up close the rendering is a deliberate pattern.

• **why it's NPR**: same thesis — *abstraction guided by structure* (**L4**) — here the abstraction lives in **how tone is quantized into marks**, not in strokes, edges, or regions. Stippling, engraving/hatching, and ASCII-art are the same move with different mark vocabularies.

• **the dial = the screen / mark set**: choice of screen element, frequency, and angle sets the style; error-diffusion vs ordered-dither sets the "texture" of the noise (blue-noise dithers look cleanest). Ties to [[Color technology]] (CMYK screen angles, moiré) and the perceptual reason it works (the eye's spatial averaging — [[Human (and animal) vision and color]]).

• **p-set lineage**: the "artistic screening" make-your-own assignment.

• a single lens fixes **one** degree of freedom (its power/focal length) but you need to satisfy **many** constraints simultaneously: bring *every* field point to focus, at *every* wavelength, across the *whole* aperture, onto a *flat* sensor.

• the thin lens **dropped** everything past the linear $\sin\theta\approx\theta$; restore the next term $\sin\theta\approx\theta-\theta^3/6$ (**third-order / Seidel optics**) and rays from one point **no longer meet at one point**. The departures are the aberrations — same approximation, dropped one term later (FUNDAMENTALS margin note). [`fig-seidel-aberrations`]

• two families: **monochromatic** (geometry of the surfaces, present even in one color — the five Seidel) and **chromatic** (because $n(\lambda)$ — dispersion). Each is *described* here; each is *corrected* in *Aberrations correction*.

#### spherical aberration

• rays through the **edge** of the aperture focus **nearer** than paraxial rays → a soft glowing core; worst wide open. (Deliberately left in for *soft-focus*, *Special optics*.) [`fig-spherical-aberration`]

#### coma

• off-axis a point images as a **comet/teardrop** smear (asymmetric) — the name *is* the comet shape.

#### astigmatism

• off-axis, tangential and sagittal line-features focus at **different depths** → you can't focus radial and tangential detail at once.

#### field curvature

• (Petzval): the sharp image lies on a **curved surface**, not the flat sensor → corners soft when the center is sharp (why "flat-field" objectives are prized).

#### chromatic aberration

• glass index varies with wavelength (**dispersion**, measured by the **Abbe number** $V_d$), so colors refract differently. [`fig-chromatic-aberration`]

• **axial (longitudinal)**: R/G/B focus at **different depths** → color halos; worst at fast apertures.

• **lateral (transverse)**: R/G/B imaged at **different magnifications** → colored fringes that **grow toward the corners** (and the everyday **purple fringing** at high-contrast edges, partly CA, partly sensor/flare). Lateral CA is essentially a **per-channel radial scaling**; axial CA is a depth-dependent blur — the two are a continuum but call for **different fixes** (*Aberrations correction*).

#### radial distortion

• magnification varies with field → straight lines bow (**barrel** / **pincushion**). Uniquely, distortion **moves no light energy** (points stay sharp, just mis-placed), so it is the **easiest to correct in software** — a **radial** remapping. [`fig-distortion-barrel-pincushion`]

• the standard parametric models used for lens-profile correction and camera calibration map undistorted radius $r_u$ and distorted radius $r_d$ (both measured from the distortion center):

• **Brown–Conrady** (radial, forward): $r_d = r_u\,(1 + k_1 r_u^2 + k_2 r_u^4 + k_3 r_u^6)$, with optional **tangential** (decentering) terms $p_1, p_2$ for off-axis-mounted elements; $k_1<0$ gives barrel, $k_1>0$ pincushion.

• **division model** (Fitzgibbon): $r_u = r_d / (1 + \lambda_1 r_d^2 + \lambda_2 r_d^4 + \dots)$ — often a single $\lambda_1$ captures even strong (wide-angle) distortion and inverts more cleanly, which is why it is popular for calibration.

#### wave effects and diffraction

• light is a **wave**; a wave through an aperture **diffracts** (spreads), so even a perfect lens images a point as the **Airy** disc + rings, angular radius $\theta \approx \lambda/D$ — a **floor** (the diffraction limit), not a defect. Bigger aperture → less diffraction.

• **the pattern *is* the aperture's Fourier transform** (squared magnitude = the diffraction-limited PSF): round hole → Airy disc; a wider hole has a narrower transform = a smaller spot (the $\lambda/D$ floor restated). Explains **sunstars** = FT of the polygonal blade opening (each blade-edge → a perpendicular streak; $n$ blades → $n$ spikes if even, $2n$ if odd). The lens computes a Fourier transform for free → ties to [[03 Basic image processing and ISP]] (Fourier) and the **MTF**.

• stopping down cures geometric aberrations but eventually raises the diffraction floor → every lens has a **sweet spot** ~2–3 stops down; past it, uniform softening.

#### vignetting

#### flare and coating

• every aberration is *described once* in *Aberrations and optical challenges*; here each is *corrected* — **in glass** (shape and combine elements, pick glasses, stop down) or **in software** (remap, rescale, deconvolve). Each design feature cancels a specific aberration, so the cures are organized by the defect they target.

• **achromat** (crown + flint doublet): pair a low-dispersion positive element with a high-dispersion negative one so the **chromatic terms cancel** at two wavelengths — the **achromat condition** $\phi_1/V_1+\phi_2/V_2=0$. [`fig-achromat-doublet`]

• **apochromat / superachromat**: correct **three (or more) wavelengths**, using **ED / low-dispersion / fluorite** glasses — for long telephotos where residual color is visible.

• **aspheric elements**: a non-spherical surface gives the designer an extra knob to kill **spherical aberration** with **fewer elements** (key to compact, fast, and **cell-phone** lenses). Sidebar: **spherical surfaces are easy to grind** (any two ground together tend to a sphere), so aspheres came late and rely on **precision molding** — which is exactly why cheap **molded plastic/glass** aspheres gave phone cameras their edge.

• **stopping down** reduces the aberrations that scale with aperture (spherical, coma, lateral color, vignetting) — the universal field fix — until **diffraction** sets the **sweet spot** (FUNDAMENTALS diffraction limit; usually ~2–3 stops down).

• **lens-profile correction** (in the ISP): from a **measured profile** per lens+focal length+aperture, software undoes **geometric distortion** (radial warp via the Brown–Conrady / division models from *Aberrations and optical challenges* — fit $k_1,k_2,\dots$ or $\lambda_1$ per lens; detailed below in *Radial distortion correction*), **vignetting** (a radial gain map — both optical *and* the FUNDAMENTALS $\cos^4$), and **lateral CA** (per-channel radial rescale). Cheap, robust, ubiquitous — the everyday face of optics-plus-computation. [`fig-lens-profile-correction`]

• **deconvolution by the measured PSF**: an aberrated lens has a known, **spatially-varying PSF**; measuring it lets you **deconvolve** to recover sharpness — the same machinery as [[Slopendium/08 Single-image computational photography|deblurring]] (Wiener / regularized inverse), here with a **calibrated** rather than unknown kernel, so it is non-blind and per-field-position. [`fig-psf-deconvolution`]

• limits: deconvolution **amplifies noise** at frequencies the lens killed (FUNDAMENTALS/deblurring L) — it sharpens what was *attenuated*, not what was *destroyed*; and a **spatially-varying** PSF means per-region kernels.

• **forward-ref**: instead of *fighting* the lens's accidental blur, **engineer** a PSF that inverts cleanly and even encodes depth — **coded aperture / wavefront coding** (Dowski & Cathey 1995) and end-to-end optics (Advanced).

• **the most software-friendly aberration** (it is a *lens* defect, not a projection one — moved here from the warping part, where keystoning lives): radial distortion **moves no light energy** (points stay sharp, only mis-placed), so it comes off almost perfectly as a **1-D remap in radius**, with **no loss of sharpness** — the cleanest cure in the catalogue. Because it is so correctable it is increasingly *designed in*: phone/compact wide-angles ship heavy **barrel** distortion on purpose (smaller, faster glass) and trust the ISP to straighten it. [`fig-radial-distortion-correction`]

• **the symptom**: world-straight lines image as **curves** — **barrel** (bow **outward**, magnification falls with radius — wide-angle), **pincushion** (bow **inward** — telephoto), or a **mustache/wavy** mix; radially symmetric about the principal point, **worst at the edges**; the extreme is the **fisheye** (so strong it is the intended projection, not a defect).

• **the model — per-radius remap**: a 1-D function of radius applied along each ray (points move only **in/out**, never sideways) — the same forward **Brown–Conrady** $r_d = r_u(1+k_1 r_u^2+k_2 r_u^4+\dots)$ ($k_1<0$ barrel, $k_1>0$ pincushion) and **division** $r_u = r_d/(1+\lambda_1 r_d^2+\dots)$ models stated in *Aberrations and optical challenges*; correcting the image **inverts** that map per radius.

• **where the parameters come from**: a **lens profile** (a per-(lens, focal length, aperture) table shipped by **Lightroom/ACR, PTLens, DxO**, keyed off **EXIF** — the same profile that undoes vignetting and lateral CA in one pass) or **plumb-line calibration** (choose the $k_i$/$\lambda$ that make imaged straight lines straight — minimize their residual curvature).

• **applying it — inverse-warp + resample**: loop over output pixels, push each back through the radial map to its source radius, and resample the input (the transport engine of [[Warping and resampling]]); the stretched/compressed edge regions carry the usual **prefilter-before-downsample** care (**L16**), and out-of-frame corners follow the boundary/crop policy. The warping part now points *back here* for this correction.

---

• restate the impossibility: beyond first order there is **no formula** that places every ray; you **search** the parameter space instead.

• **parameters (the unknowns)**: each surface's **radius of curvature**, the **glass** of each element (index *and* dispersion — a discrete catalog choice), element **thicknesses** and **air-gap spacings**, **aspheric** coefficients, and the **stop** size/position.

• **objective (the merit function)**: a single number summing **residual aberration** over a grid of **field angles**, a set of **wavelengths** (e.g. C, d, F lines), and a sampling of **aperture/pupil** zones — typically RMS spot size or wavefront error — plus **hard constraints** (target $f$, total length, positive edge thickness, vignetting limits).

• "**the big loss integral**": conceptually, integrate the blur over *all the conditions the lens must satisfy at once* (every field point, color, aperture position) and drive it down — the discrete merit function is that integral, sampled.

• evaluate a candidate design by **exact ray-tracing**: launch many rays from each field point through a grid of pupil points, refract them by **Snell's law at every surface** (the full non-linear law, not the paraxial one), and see where they land.

• a **spot diagram** plots those landing points at the image plane for one field point and color: a tight cluster = sharp; a smeared blob = aberrated. Different colors land in different clusters (chromatic). This *is* the lens's PSF, sampled by rays. [`fig-spot-diagram`]

• the optimizer shrinks these spots across all fields/colors simultaneously. [`fig-merit-function-loop`, `fig-rms-spot-vs-field`]

• **the old-fashioned way**: designers traced rays **by hand** (and with logarithm tables, then mechanical/early electronic calculators), iterating curvatures laboriously — a few rays a day. The classic doublets/triplets were found this way.

• **today**: optical-design software (**Zemax OpticStudio**, **CODE V**, OSLO) ray-traces thousands of rays per evaluation and runs **damped-least-squares** optimization over the parameter vector, plus **global** search to escape local minima and **tolerancing** to keep the design manufacturable.

• **how people actually use it** (workflow, not a black box): start from a **starting point** (the program's library, a scaled existing lens, or a competitor's **patent** prescription) — never a blank sheet; set up the *problem* — field points, wavelengths + weights, f-number, then mark which parameters are **variables** and which are pinned by **solves** (auto-recompute one number to hold focal length / back focus); pick the **default** RMS-spot/wavefront merit + hand-added **operands** (cap distortion, edge thickness, length, glass cost); run **local** damped-least-squares (seconds), read **spot diagrams / ray fans / MTF**, nudge weights, re-run — a tight **interactive loop**, optimizer does arithmetic, human supplies judgment; when stuck, unleash **global** search (Zemax "Hammer", CODE V global synthesis) for hours/overnight; then **tolerance** + export to mechanical CAD / stray-light. The craft moved *up a level*: from tracing rays by hand to **steering** an optimizer.

• it is the same machinery as the rest of the book — forward model + merit/loss + iterative optimizer — applied to glass.

• the merit function lives under hard **constraints**, and the axes fight: **speed** (low f-number) vs. **aberration control**; **size/weight** vs. **correction**; **cost** (number of elements, exotic glass, aspheres) vs. **sharpness**; **zoom range** vs. **everything**. You buy one by spending another. [`fig-lens-design-tradeoff`]

• this is exactly why **computation** is attractive: let the optics be *good enough and cheap*, and **correct the rest in software** (lens-profile correction, deconvolution — *Aberrations correction*), shifting the operating point on the Pareto front.

• **forward-ref — end-to-end optimization**: make the *whole pipeline* (lens + sensor + reconstruction network) differentiable and optimize the **merit of the final image**, not of the optics alone — the optics may then look "wrong" by classical metrics yet give better pictures (Advanced part: differentiable / computational optics, coded apertures, wavefront coding).

• **the error sources**: per-surface **radius** error, **thickness / airspace** error, glass **index & dispersion** (melt-to-melt) variation, **wedge**, surface **irregularity** (departure from the test sphere), and assembly **decenter / tilt / roll** of each element

• **sensitivity vs statistics**: a **sensitivity analysis** ranks which perturbations hurt most (the tight tolerances that drive cost); a **Monte-Carlo** draws many random as-built lenses and predicts the **performance distribution** and the manufacturing **yield**

• **compensators**: leave a few adjustments free at assembly — refocus, set one **airspace**, or shim/center an element — to *recover* performance, so tolerances elsewhere can loosen (cheaper)

• **the punchline**: nominal performance is the easy part; **as-built** performance and **yield** decide cost and feasibility. Loosen tolerances and lean on **computational correction** (lens-profile correction, deconvolution — *Aberrations correction*) to claw back the residual — another reason cheap optics + computation wins.

---

• **principal planes** $H, H'$: the two planes from which object/image distances and focal length are measured; with them, the thin-lens equation $1/f=1/u+1/v$ holds for the whole thick system. **Nodal points** (where a ray crosses the axis undeviated) coincide with the principal points in air — the pivot for "no-parallax" panorama heads. [`fig-principal-planes-pupils`]

• **entrance & exit pupils**: the images of the aperture stop seen from object / image space. The **entrance pupil** is the opening that actually gathers light, so the true f-number is $N = f/D_{\text{EP}}$ — *not* $f$ over the front-element diameter. Pupil position governs perspective of the bokeh and the **no-parallax point**.

• **f-number** (recap from FUNDAMENTALS) sets light-gathering and DoF; here it is pinned to the entrance pupil. **Vignetting** clips the pupil off-axis (next section's cat's-eye bokeh and the optical part of corner darkening).

• **T-stop**: the f-number assumes 100% transmission; real glass and coatings lose some, so cinema lenses are marked in **T-stops**, $T = N/\sqrt{\tau}$ with transmittance $\tau$, giving the *actual* exposure. Ties to *Glare suppression* (coatings raise $\tau$) and FUNDAMENTALS exposure.

• **element count today**: modern zooms and fast primes run **10–20+ elements** in several groups (with aspheres and special glasses) — a direct consequence of stacking corrections; phone "lenses" are 5–8 tiny molded-plastic/glass elements. The count is the visible cost of cancelling everything in glass instead of software.

---

• **shift**: translate the lens parallel to the sensor to control **perspective** (keep verticals parallel — architecture) and to **stitch** without parallax; the image circle must be large enough to cover the shifted frame. [`fig-tilt-shift-perspective`]

• **tilt**: rotate the lens so the **plane of focus tilts** away from parallel-to-sensor. By the **Scheimpflug condition**, the lens plane, image plane, and plane of sharp focus all meet along one line — so you can lay the focal plane along a tabletop (everything sharp) or slice a thin wedge across a scene (the "miniature-faking" look). [`fig-scheimpflug-tilt`]

• ties to *Depth of field*: tilt reshapes the in-focus region from a slab into a **wedge**; it does not add DoF, it **reorients** it.

• a normal lens targets **rectilinear** projection ($r=f\tan\theta$) — straight lines stay straight — which **blows up** past ~100° field. A **fisheye** instead adopts a projection that *bends* straight lines to fit a huge field (often **180°+**) onto the sensor: **equidistant** $r=f\theta$ or **equisolid-angle** $r=2f\sin(\theta/2)$. [`fig-fisheye-projection`]

• the barrel "distortion" is **the intended mapping**, not an aberration — and it is **invertible in software** to rectilinear (at the cost of stretched corners), the cleanest example of optics-and-computation splitting the work.

• **catadioptric** = mirrors **and** lenses. Folding the path with mirrors makes a **compact long telephoto** (mirror lens) and is naturally **achromatic** (reflection has no dispersion). The **central obstruction** gives a **ring-shaped pupil**, so out-of-focus highlights render as **doughnuts** — the signature mirror-lens bokeh (cross-ref *Depth of field*). [`fig-catadioptric-mirror-lens`]

• **telescopes**: reflectors (Newtonian, Cassegrain) and catadioptrics (Schmidt–Cassegrain, Maksutov) scale the same idea to large apertures; (space) telescopes also motivate the **glare/baffling** chapter. [`fig-telescope-types`]

• **the constraint**: a phone body is only a few millimetres thick, so a lens whose axis runs perpendicular to the screen cannot be physically long — capping the focal length and thus the telephoto reach.

• **the fix**: a **prism or mirror** placed just behind the entrance window **bends the optical axis ~90°**, so the lens groups lie *sideways* along the phone's width or length, where there is room for a long focal length; the sensor sits parallel to the phone's back. Because the path is "folded" the design is called a **periscope** or **folded-optics** lens. [`fig-catadioptric-mirror-lens` (cross-ref, same folding idea)]

• **what it buys**: a long effective focal length — ~5×–10× optical zoom / ~100–250 mm equivalent — in a thin body, without the protruding bump a straight long-lens design would require.

• **mechanics**: moving internal groups give continuous or stepped zoom; **OIS** is commonly implemented by *tilting or shifting the prism* (or a group) rather than floating a lens element — a compact, low-power actuator.

• **tetraprism (Apple, iPhone 15 Pro Max)**: folds the path **multiple times** with a tetrahedral prism, lengthening the effective optical path further within the same device width.

• **history**: popularized by the **Huawei P30 Pro (2019)**, then adopted by Samsung (Galaxy S20 Ultra and successors) and Apple; the reason phones acquired genuine long-reach telephoto. Cross-ref: the **telephoto design** in [[A short bestiary of classic designs]] compresses *physical length* relative to focal length by placing principal planes; the periscope instead *reorients* the whole long lens so its length runs parallel to the phone face — two independent strategies for compact telephoto, usable together.

• **trade-offs**: the small front window and folded path limit the **maximum aperture** (less light than a rear-camera main module); each periscope module covers a fixed or narrow zoom range; prism alignment and cost add manufacturing complexity.

• a cylindrical-element system **squeezes the image horizontally** (classically 2×) so a wide aspect ratio fits a narrower frame, then a matching de-squeeze restores it on projection. [`fig-anamorphic-squeeze`]

• side effects that became a *look*: **oval bokeh** and **horizontal (blue) flare streaks** — an aesthetic now imitated computationally.

• **macro**: images at **life-size or larger** ($m\ge 1$), where the thin-lens conjugates run the *other* way (image distance large), DoF is razor-thin (cross-ref *Depth of field* — **focus stacking** lives there; [Stanford focal-stack paper](https://graphics.stanford.edu/papers/focalstack/focalstack.pdf)), and special **flat-field** correction matters.

• **microscope objectives** and **telescope objectives** are the same compound-lens optimization pushed to extreme magnification / aperture, with their own corrections (apochromatic, flat-field). The high-NA microscope objective images a tiny subject at a short **working distance**, in either a **finite** (fixed tube length) or modern **infinity-corrected** (parallel "infinity space" + a separate tube lens) configuration. [`fig-microscope-objective`]

• **soft-focus** portrait lenses deliberately leave **under-corrected spherical aberration**, overlaying a sharp core with a glowing halo — a *wanted* aberration. [`fig-soft-focus`]

• **apodization**: a radially-graded (smooth-edged) pupil that **softens the bokeh transition** (smooth-edged defocus disks, no hard ring) — shaping the PSF on purpose, a hardware cousin of computational bokeh. [`fig-apodization-pupil`]

• *(throughline to the Advanced part: "keep the phase / slope, drop the thickness.")*

#### Fresnel lenses

• a **Fresnel lens** keeps only the part of a lens that actually bends light — its **surface slope** — and throws away the bulk glass: slice the lens into concentric annular zones and **collapse each to one thin sheet**, so the smooth profile becomes a set of **sawtooth rings** (a ring of tiny prisms) with the same local refraction as the original curve.

• **why**: dramatically **thinner, lighter, cheaper** than an equivalent thick lens — at the cost of **image quality** (scattering and diffraction at the ring discontinuities, the visible concentric-ring artifacts), so it is used where thinness / weight / cost beat sharpness.

• **history & uses**: invented by **Augustin Fresnel** for **lighthouses** (a huge aperture without an unliftable blob of glass); today in **condenser / projection optics**, **screen & page magnifiers**, **solar concentrators**, **vehicle brake lights**, thin illumination optics, and the **Fresnel/PF telephoto** elements in modern camera lenses.

• **the throughline**: it is the first step from "shape the glass" toward **encoding the lens as a thin structure** → diffractive optics, kinoforms, and ultimately **flat optics / metalenses** (Advanced); the same "keep the phase, drop the thickness" idea recurs in **wavefront coding**.

#### Diffractive elements

• bend light by **diffraction** off fine surface structure rather than bulk refraction; their dispersion has the **opposite sign** to glass, so a diffractive element can **correct chromatic aberration** in a hybrid lens (Canon DO, Nikon PF). Thin, light, but flare-prone at the rings. [`fig-grin-vs-metalens`]

• the canonical diffractive element is the **diffraction grating** — a periodic array of slits/grooves of period $d$. Wavelets from adjacent periods reinforce only where their path difference is a whole number of wavelengths, the **grating equation $d\sin\theta_m = m\lambda$**, so incident light splits into discrete **orders** $m=0,\pm1,\pm2,\dots$ at angles $\theta_m$. Because $\theta_m$ **grows with $\lambda$**, a grating **disperses** white light into a spectrum — the heart of the **spectrometer** — dispersing in the *opposite* sense to a prism (prisms bend blue most; gratings bend red/long-λ most), and the same physics behind the rainbow sheen of a **CD/DVD**. 🖱️ **Interactive (web edition):** a live 2-D **FDTD wave simulation** — a plane wave hits a multi-slit grating and breaks into orders along the $d\sin\theta=m\lambda$ directions; **sweep the wavelength and watch the orders fan wider** (longer λ → larger deviation), or change the period. Absorbing (sponge + Mur) borders, so nothing reflects off the window edges. Companion to the single-slit `fig-diffraction-wave-sim` (FUNDAMENTALS). [`fig-diffraction-grating-sim`; interactive, FDTD]

#### Mirrors (cross-link)

• see *Mirrors: catadioptric and reflecting systems* above (reflection is the other way to avoid dispersion).

#### GRIN (gradient-index)

• the index **varies continuously inside** the glass, so a ray bends along a *curved* path even through flat surfaces — focusing with a homogeneous-looking slab. Used in fiber coupling, endoscopes, photocopier rod arrays, and the eye's own lens. [`fig-grin-vs-metalens`]

#### Metalenses (forward-ref)

• a flat **sub-wavelength nanostructure metasurface** imposes the lens's phase profile with essentially **zero thickness** — the endpoint of the Fresnel→diffractive→flat-optics arc. Full treatment in the **Advanced** part; placed here as the conceptual destination.

---

• focusing moves the **lens (or a focusing group, or the sensor)** to set the image distance $v$ so the subject's conjugate falls on the sensor; an object at infinity sits at $v=f$, nearer objects push $v$ back (FUNDAMENTALS). [`fig-focus-conjugate-move`]

• **manual focus aids**: ground-glass/split-prism, the **rangefinder** (a coincidence/disparity device — cross-ref FUNDAMENTALS stereo), and modern **focus peaking** / magnified live-view.

• **focus-by-wire**: in many AF lenses the focus ring is *not* mechanically linked — it sends an electronic signal to a motor (stepper / ultrasonic **USM** / linear voice-coil), enabling fast, quiet, programmable, and lens-data-aware focusing.

• **focus breathing**: as you refocus, a lens's **focal length / field of view subtly changes** — the image appears to **zoom slightly** as the focus pulls (because the focusing group's motion also shifts the effective $f$). An artifact in two regimes: in **video** it shows as a visible "breathing" during **focus pulls** (distracting in cinematography), and in **focus stacking / depth-from-defocus** it means consecutive frames **don't register** (different magnification → must be re-aligned, cross-ref *Depth of field* §focus stacking and part 08's focal-stack chapter, which calls it out). Modern **cine lenses** are designed to be breathing-free, and some cameras/software **correct** it digitally.

• premise: a sharp image has **more high-frequency energy** than a blurred one. Maximize a **contrast metric** (sum of squared gradients / high-pass energy) over the focus position — gradient *ascent* on the sharpness hill. [`fig-contrast-af-curve`]

• strength: needs **no special hardware**, uses the imaging sensor itself, very accurate at the peak.

• weakness: the metric gives **no direction** — the camera must **hunt** (step, observe, maybe reverse), so it is slower and visibly racks back and forth; struggles in low contrast / low light.

• compare light from **two halves of the exit pupil**: in a dedicated AF sensor a **separator (secondary) lens** re-images the two pupil halves onto **two line sensors**. The **lateral offset** between the two patterns is a **signed** measurement of defocus — its **sign gives the direction**, its **magnitude gives how far** — so the lens can be driven to focus in **one move**, no hunting. [`fig-pdaf-principle`]

• this is literally **stereo inside the lens**: the baseline is the pupil split, and the offset is a disparity proportional to defocus (cf. FUNDAMENTALS $Z=fB/d$).

• classic **SLR** implementation: a partly-transparent main mirror + sub-mirror diverts light down to a dedicated PDAF module — fast, but a *separate* optical path needing calibration (front/back-focus).

• **sidebar — why the reflex mirror is disappearing** (synthesis of the author's 2018 *"DSLR will probably die"* essay, see [[Blog Index]]): the mirror's old benefits (optical TTL view, dedicated fast PDAF) shrink as EVFs improve and on-sensor PDAF closes the gap; its costs do not — added mechanism + per-body/lens calibration, larger body, the **retrofocus tax** on wide/fast lenses (cross-ref bestiary), and it **occludes the sensor while focusing** so no pre-shot scene analysis (face/eye); for video the mirror stays up, forfeiting all advantages. Reflex → mirrorless; AF migrates onto the main sensor.

• **masked / phase-detect pixels**: scatter pixels across the imaging sensor whose left or right half is masked, recovering the two pupil views *on the main sensor* — no separate module (enables mirrorless/live-view PDAF). [`fig-dual-pixel-af`]

• **dual-pixel AF**: split **every** photosite into **two half-photodiodes under one microlens** (reusing the FUNDAMENTALS microlens stack) — each pixel is its own phase-detect pair, giving dense, fast AF *and* a low-baseline disparity map (small depth signal usable for portrait-mode and refocus).

• **the two are different strategies**: **depth from focus** *sweeps* the focus through its range and picks, per pixel, the **setting that maximizes sharpness** — many shots, no blur model needed (the focus position *is* the depth). **Depth from defocus** instead estimates depth from the **amount of blur** in just **one or two** shots, given a **known blur (PSF) model** of how defocus scales with depth and aperture — far fewer frames, but it leans on the model.

• **depth from focus**: sweep focus, find for each pixel the position of **maximum sharpness** → depth (also the basis of **focus stacking**, *Depth of field* — the focal-stack sharpness map is this same computation, [Stanford focal-stack paper](https://graphics.stanford.edu/papers/focalstack/focalstack.pdf)).

• **depth from defocus**: from a few images at different focus/aperture, infer per-pixel **blur radius** → depth, with far fewer shots (Pentland 1987; Subbarao). The defocus PSF is the bridge to deconvolution (*Aberrations correction*). [`fig-depth-from-defocus`]

• **depth-from-defocus *as autofocus* — Panasonic DFD (Depth From Defocus)**: rather than build a depth map, use defocus to **drive focus**. With a stored model of how a given lens's blur (PSF) changes with focus, the camera compares **two frames at slightly different focus** and infers the **direction *and* amount** of defocus — a PDAF-like, near-single-step cue from an ordinary **contrast-detect** sensor with **no phase pixels**. It rides on CDAF (confirms at the sharpness peak) and is strong for video; its weakness is the reliance on a **per-lens blur profile** (and on the subject holding still between the two frames).

• **subject / eye AF (learned)**: a detector finds **faces, eyes, animals, vehicles**, and tracks them, then drives PDAF to the chosen point — AF's *decision* layer is now a recognition problem (cross-ref the learned-detection chapter). Other rangefinding aids historically/peripherally: active **IR / ultrasonic** (early Polaroid SONAR) and **LiDAR/ToF** assist in phones.

• **why autofocus fails**: stars are **dim point sources at infinity** with almost no contrast for CDAF and too little light for PDAF — so astro focus is essentially **manual** and demands *more* precision than daytime (a star is the ultimate test of focus; the circle of confusion must be tiny)

• **you can't just rack to infinity**: most lenses **focus past infinity** (for thermal expansion / AF overshoot), so the "∞" hard stop is *not* in focus — you must focus empirically each session, and **refocus as the temperature drops** (the barrel contracts)

• **the practical methods**: focus on a **bright star** with **magnified live view**; or use a **Bahtinov mask** — a grid placed over the aperture whose **diffraction spikes** form an X with a central spike that is centred *only at exact focus* (a precise, eyeballable null); software **FWHM / HFD** metrics on a star automate the same idea, and **motorized focusers with autofocus routines** (minimize star size vs focuser position — contrast-AF, done deliberately and slowly) are standard on deep-sky rigs

• cross-ref: the daytime AF families above (why they don't transfer); *Depth of field* (tiny CoC); FUNDAMENTALS Focus (the practical operator's view)

---

• briefly restate the circle of confusion, the near/far limits, **hyperfocal** $H=f^2/(Nc)$, and the three knobs (smaller aperture, shorter $f$, farther focus all deepen DoF) — then **point to FUNDAMENTALS** for the full similar-triangles derivation and the framing-invariance result. [`fig-dof-coc`]

• restate the load-bearing caveat: **defocus is not a 2-D convolution** (it is depth- and occlusion-dependent) — the reason naive portrait-mode fakes fail and depth-aware/light-field methods are needed (forward-ref Advanced).

• the out-of-focus highlight is the **image of the aperture stop**, so its **shape = the aperture's shape**: a round iris → round disks, a few straight blades → **polygons** (pentagons, heptagons). [`fig-bokeh-shapes`]

• **disk structure** sets "good" vs. "bad" bokeh: a **bright-edged** disk (from over-corrected spherical aberration) looks **busy/nervous**; a **uniform or soft-edged** disk (under-corrected, or apodized — *Special optics*) looks **creamy**. The *whole* point of a portrait lens is the disk profile, not just resolution.

• **cat's-eye / lemon bokeh**: toward the frame edge the pupil is **clipped by the lens barrel and front/rear elements** (optical vignetting), so off-axis defocus disks are **cut to lemon shapes**, swirling toward the corners; stopping down restores round disks (and reduces corner vignetting). [`fig-cats-eye-vignetting`]

• ties to *Compound lenses* (pupil/vignetting) and *Special optics* (mirror-lens doughnuts, anamorphic ovals, apodizers).

• **focus stacking**: capture a **stack** at stepped focus distances, then composite the **sharpest pixels** from each into one **all-in-focus** image — the standard fix where DoF is hopeless (**macro**, product, landscape near-to-far) ([Stanford focal-stack paper](https://graphics.stanford.edu/papers/focalstack/focalstack.pdf)). [`fig-focus-stacking`]

• requires per-pixel **sharpness selection** + alignment (focus *breathing* changes magnification between shots → must register); a multi-image-fusion problem (cross-ref the fusion/stacking machinery).

• **the sharpness map is depth-from-defocus**: estimating, per pixel, *which slice of the stack is sharpest* is exactly **depth from focus/defocus** (cross-ref *Focus, autofocus*) — the focus position of maximum sharpness corresponds to that pixel's **depth**, so a focal stack yields an all-in-focus image **and** a depth map for free.

• contrast with *increasing* DoF by stopping down — which costs light and eventually **diffraction** softens everything (FUNDAMENTALS): stacking beats the diffraction wall.

• **shallow on purpose**: open up / step closer / longer lens to *magnify* the subject — recall DoF is set by **magnification**, not focal length per se (FUNDAMENTALS framing-invariance).

• **fake / computational DoF (portrait mode)**: estimate a **depth map** (dual-pixel disparity from *Focus*, stereo, or learned mono-depth) → apply **depth- and occlusion-aware** blur with a chosen disk shape. Honest about why it is hard (hair, edges, transparency) — full treatment forward-referenced to the Advanced computational-camera part; here we only set up *why* a 2-D blur is wrong and *what* a depth map buys.

• **reminder — defocus is NOT a single 2-D convolution**: the blur **kernel size varies with depth** (a far point blurs more than a near one), and **occlusion boundaries break shift-invariance** (a sharp foreground edge bleeds onto a blurred background, not symmetrically) — so no one uniform kernel applies. This is exactly why portrait-mode background blur needs a **depth map** rather than a flat Gaussian (cross-ref the *depth-of-field geometry* recap above and FUNDAMENTALS [[Manuscript/Draft — 02-02 Image formation and linear perspective|Image formation §Depth of field]]).

• **sidebar — coherent imaging (Barbastathis): out-of-depth is *black*, not blurry**: everything above is **incoherent** imaging, where an out-of-focus object still contributes light (a blur disk). George Barbastathis (MIT) notes that **coherent** imaging built around a *prepared* depth can do something qualitatively different — off-depth objects contribute **no light at all**, so they come out **black, not blurry** (incoherent imaging adds positive *intensities*, so every depth contributes; coherent imaging adds *complex amplitudes*, which can interfere destructively and cancel). Reframes DoF as a hard **on/off depth gate** rather than a graceful blur.

---

• at **every** air–glass surface a few percent of light **reflects** (Fresnel, ~4% uncoated); a stray ray can reflect **twice** and reach the sensor as a **ghost** — a discrete, often aperture-shaped bright blob, usually mirrored across center from a strong source. [`fig-lens-flare-ghosting`]

• many weak scatters and reflections sum into **veiling glare** — a low-frequency **wash that lifts the blacks** and crushes contrast (worst with a bright source just outside the frame, or shooting into the sun/backlight).

• **why compound lenses make it worse**: $N$ surfaces give $O(N^2)$ inter-reflection pairs — a 15-element lens has dozens of surfaces, so without coatings it would be hopeless (cross-ref *Compound lenses* element count, and the T-stop transmittance loss).

• **flare as an aesthetic (cinematography)**: lens flare was long considered a **flaw / a no-no** in cinematography (the whole point of coatings and hoods is to *kill* it), but **modern films deliberately court it** for a dramatic / realistic, "you're really there" effect — **J.J. Abrams' anamorphic streaks** are the cliché. It also **conveys very high contrast / the presence of a bright source**: flare reads as "that light is *intense*," a cue the eye trusts, so a little flare can sell a bright window or the sun better than a clean frame. (An aesthetic now also **faked computationally** — cross-ref anamorphic flare, *Special optics*.)

• **not the same as a sunstar**: lens **flare/ghosts** are **inter-surface reflections**; the **star / sunstar** pattern radiating from a bright point is a **diffraction spike** — diffraction at the **aperture blades** (the FT of the blade polygon, cross-ref *Aberrations*/wave effects), present even in a single perfectly-coated element. Different physics, often confused.

• **anti-reflection coatings**: a thin transparent film of index $n_c=\sqrt{n_1 n_2}$ and thickness **λ/4** makes the reflections off its two faces **interfere destructively**, cancelling the surface reflection over a band; **multi-layer** stacks broaden it to the whole visible range (the colored sheen of modern coatings). This dropped per-surface reflection from ~4% toward ~0.1–0.5%, which is what *made* many-element lenses practical. **Cross-ref the FUNDAMENTALS lens-coating history.** [`fig-ar-coating-interference`]

• **sidebar — the coating, Zeiss, and the spoils of war**: Smakula's 1935 Zeiss-Jena patent (DE 685767), classified as a military secret (the red "T"/Transparenz mark, ancestor of T\*); independently re-invented in the open by Strong 1936 / Blodgett 1938; then the 1945 Allied seizure of German patents (FIAT microfilming, US Alien Property vesting, 1946 London Agreement) and the Jena↔Oberkochen split of Zeiss scattered it. Lesson: it spread because it was discovered in the open more than once, not because one patent fell.

• **everyday encounters with the same trick**: these are the **same coatings on eyeglasses** (anti-reflective lenses — fewer ghost reflections, less glare for the wearer and for people looking at you) and the **anti-glare / AR coatings on displays and picture-frame glass** (museum / "non-reflective" glass) — the quarter-wave **interference** trick is all around you, not just inside camera lenses.

• **lens hood**: blocks off-axis light (a bright source *outside* the frame) from striking the front element at all — the cheapest, most effective anti-flare tool. [`fig-lens-hood-baffles`]

• **internal baffles, blackened edges, light traps, and anti-reflective barrel paint**: stop grazing rays from skipping down the inside of the barrel; critical in **(space) telescopes**, where stray sunlight must be baffled to image faint targets (cross-ref *Special optics* telescopes). [`fig-lens-hood-baffles`]

• model veiling glare as the true image **convolved with a broad glare PSF** (plus discrete ghosts); **measure** that glare spread function and **deconvolve / subtract** it to **recover contrast** — same PSF-deconvolution toolbox as *Aberrations correction*, applied to stray light. [`fig-veiling-glare-psf`]

• **dehaze-style** contrast recovery (cross-ref [[Slopendium/08 Single-image computational photography|dehazing]]) and **lens-specific flare/ghost removal** (learned, since ghost patterns are lens-specific). Important for **HDR** capture, where veiling glare from bright sources otherwise corrupts the dark-end measurements (Talvala 2007). Forward-ref Advanced for learned deflare.

---

• hand tremor **rotates** the camera by small angles during the exposure; the image shifts by $\approx f\,\Delta\theta$, so the **smear grows with both shutter time and focal length** (the old "1/focal-length" minimum-shutter rule). [`fig-handshake-motion-blur`]

• restate the **blur budget** from FUNDAMENTALS: you want a short shutter to freeze shake, but that costs light → wider aperture (less DoF) or higher ISO (more noise). Stabilization **relaxes** that budget by letting you use a *longer* shutter without smear.

• **key scope limit (state up front)**: stabilization fights **camera** motion only. A **moving subject** still blurs — no amount of IS helps; that is **motion deblurring** (forward-ref Advanced/Motion).

• **sensing**: **MEMS gyroscopes** (and accelerometers) measure the camera's angular velocity in real time; a controller computes the counter-motion and drives an actuator in a **feedback loop** to null the image displacement. [`fig-gyro-feedback-loop`]

• **lens-shift OIS**: a **floating lens group** moves **laterally** (voice-coil actuators) to **steer the image back** onto its original spot. Pros: optimized per lens, the viewfinder/AF sees the stabilized image; corrects shift well. (Canon IS / Nikon VR.) [`fig-ois-vs-ibis`]

• **sensor-shift / IBIS**: the **sensor** rides a magnetic stage and **translates (and rotates/roll)** to follow the image. Pros: works with **any** lens (even un-stabilized/adapted), can correct **roll** (a 5th axis) and pitch/yaw translation; combines with lens OIS ("dual IS"). [`fig-ois-vs-ibis`]

• **how many stops**: modern systems buy **~3–7 stops** of slower shutter; the limit is residual low-frequency drift, actuator travel, and (for IBIS) keeping the moved sensor inside the lens's image circle.

• **electronic / digital stabilization**: **crop** a margin and **warp each frame** to cancel inter-frame motion (video) — no moving parts, but **costs resolution/field of view** and works *between* frames, not *within* a single exposure (so it doesn't reduce per-frame motion blur the way optical IS does). Phones combine gyro data + optical flow for very stable video. [`fig-digital-vs-optical-stab`]

• **rolling-shutter** interaction: on CMOS, fast motion + electronic stabilization must also undo **rolling-shutter** skew (cross-ref Motion/Video).

• **the computational-photography flip**: instead of one long stabilized exposure, take a **burst of short, blur-free frames** and **align + merge** them — short shutters freeze shake, alignment removes the offset, averaging recovers SNR ($1/\sqrt{N}$). This **handheld burst** approach (cross-ref multi-frame denoising / super-res) is how phones get sharp low-light shots without bulky IS — the software counterpart to optical stabilization (forward-ref Advanced).

• **cornea + crystalline lens** form the objective (most of the power is the **cornea**); the **iris/pupil** is the aperture (~f/2.1 dilated to ~f/8 constricted); the **retina** is the sensor (the **fovea** is the sharp centre); **accommodation** — the ciliary muscle reshaping the soft lens — is the eye's **autofocus** (cross-ref *Focus*)

• like any lens the eye has **aberrations** and a diffraction-limited **PSF**; visual acuity ("20/20") is its resolution spec — the optical chain of this whole part, in wetware

• **myopia (near-sighted)**: eye too long / too powerful → image focuses **in front of** the retina; distant objects blur → corrected with a **negative (diverging)** lens. (Now near-epidemic, linked to near-work and lack of outdoor light.)

• **hyperopia (far-sighted)**: eye too short → focus falls **behind** the retina → corrected with a **positive (converging)** lens

• **astigmatism**: the cornea/lens is not rotationally symmetric (**toric**) → different power in different meridians, orientation-dependent blur → corrected with a **cylindrical** lens — literally this part's *astigmatism*, in the eye

• **presbyopia**: with age the lens stiffens and **accommodation fails** → can no longer focus up close (reading distance); near-universal after ~45 — the driver of the next section

• aside — **the pinhole effect**: squinting (or **pinhole glasses**) sharpens by stopping the pupil down (more DoF), at the cost of light (FUNDAMENTALS — DoF)

• **spectacles** (the original computation-free fix), **contact lenses** (sit on the cornea → wide field, no prism/distortion), **intraocular lenses (IOLs)** (replace the lens, e.g. cataract surgery), and **refractive surgery** (**LASIK / PRK** reshape the cornea — surgically editing the prescription)

• eyewear is **applied lens design**: anti-reflection **coatings**, **high-index** glass for thinner lenses, **aspheric** surfaces to cut aberration, **photochromic** (auto-darkening) and **polarized** sunglasses (cut glare — the same physics as the camera polarizer)

• **the pinhole trick — a smaller aperture sharpens any eye.** Squinting, looking through a pinhole, or **pinhole (stenopeic) glasses** improve vision for *any* refractive error with no prescription: shrinking the entrance aperture keeps only the near-axial rays, so the retinal **blur circle collapses** — the eye's version of *stopping a lens down for more depth of focus*. The catches are a camera's: the image gets **dim**, and a tiny enough hole is **diffraction-limited**.

• **conversely, a wide-open pupil makes things worse.** In dim light (or under dilation) the pupil opens to admit marginal rays — and exactly as a wide camera aperture gives a **shallow depth of focus**, it also exposes the eye's own **aberrations** (spherical aberration, "night myopia," halos/starbursts around lights). Night acuity drops for the same reason a lens is softest wide open and sharpest stopped down; stopping the eye down (bright light, or a pinhole) fixes both the defocus *and* the aberrations at once.

• **pinhole as sunglasses.** Because it also blocks most of the light, a pinhole or narrow slit doubles as a glare-cutting **dark aperture** — sharpening *and* dimming together. Inuit **snow goggles** (a slit cut in bone/antler/wood) are exactly this, used for millennia against snow glare.

• **side bar — a short history of correcting human vision.** Magnifying **reading stones** (plano-convex glass laid on text) appear by the 11th century; the first wearable **spectacles** in Italy around **1286** (convex lenses for presbyopia and hyperopia), with concave lenses for myopia following in the 15th–16th centuries. **Bifocals** are credited to Benjamin **Franklin** (1784). Practical **contact lenses** date to the late 19th century (glass scleral lenses, Fick 1888), turning soft/hydrogel in the 1960s–70s. **Intraocular lenses** begin with Harold **Ridley** (1949). **Refractive surgery** — radial keratotomy, then **LASIK / PRK** — reshapes the cornea from the 1980s–90s. The thread: each step moves the correction **closer to the eye** — on the nose, on the cornea, inside the eye, then carved into the cornea itself.

• the problem: presbyopia needs **different power for near vs far**, but a simple lens has **one** focal length

• **bifocals / trifocals** (Franklin): two or three zones with a visible line (near segment low) — abrupt jump and an "image-jump" artifact

• **progressive (varifocal) lenses**: a **continuous gradient** of power top→bottom, no line — at the cost of **peripheral distortion** and a narrow usable corridor

• **adjustable / autofocus (AF) glasses** — the *active* alternative: **focus-tunable lenses** whose power changes on demand, so one pair spans the whole range:

• **fluidic / Alvarez lenses** — manually pumped or slid to set power (adjustable-focus eyewear for the developing world — Josh Silver)

• **liquid-crystal / electronic lenses** — a voltage changes the LC retardance → an electrically **switchable or tunable** add-power (a programmable **phase** element — the same family as the **LC-SLMs** of the Advanced part); paired with an **eye-tracker / rangefinder** they *autofocus* to wherever you look (Deep Optics / "32°N")

• the throughline: **AF glasses are accommodation re-implemented in hardware** — the eye's lost autofocus restored by a tunable lens, the wearable cousin of camera autofocus and of the phase modulators in the Advanced part

• **the observation**, before any math: point a tripod-mounted camera at a static, evenly-lit grey patch and shoot a burst. Each pixel *should* read one constant value; instead it **fluctuates** frame to frame, and across the patch (it should be flat) it's grainy. Two faces of the same thing — noise as **fluctuation over time** and **fluctuation over space when the answer should be constant**. [`fig-averaging-N-frames`]

• **the naive algorithm** is a one-liner: sum the frames, divide by $N$.

```

mean = zeros_like(imSeq[0])

for im in imSeq: mean += im

mean /= len(imSeq)

```

Visibly, 1 → 3 → 5 → 45 frames: the grey patch's histogram tightens, dark areas clean up. *Why* — and *how fast*? [`fig-averaging-N-frames`]

• **the model** (the load-bearing assumption): each pixel's value in frame $i$ is the **true signal plus noise**, $X_i = \mu + n_i$, where the noise is **zero-mean** ($\mathbb{E}[n_i]=0$ — averaging is only honest if the noise has no DC offset) and **independent** across frames (frame $i$'s noise tells you nothing about frame $j$'s — true for shot & read noise, *false* for fixed-pattern noise; flag this now, fix it in *Calibration*). Each has variance $\operatorname{Var}[n_i]=\sigma^2$.

• **the two tools** (state them as the reusable algebra): for a random variable, scaling squares the variance, $\operatorname{Var}[kX]=k^2\operatorname{Var}[X]$; and for **independent** variables, **variance of the sum is the sum of variances**, $\operatorname{Var}[\sum_i X_i]=\sum_i\operatorname{Var}[X_i]$ (independence is what kills the cross terms / covariance). [`fig-iid-variance-adds`]

• **the derivation** (two lines): the average is $\bar X=\tfrac1N\sum_i X_i$. Its **mean is the true value** — $\mathbb{E}[\bar X]=\tfrac1N\sum_i\mathbb{E}[X_i]=\mu$ — so averaging is **unbiased**: it doesn't distort the signal. Its **variance**: pull out the $\tfrac1N$ (squares, by the first rule), then add (by independence):

$$\operatorname{Var}[\bar X]=\frac{1}{N^2}\operatorname{Var}\Big[\sum_i X_i\Big]=\frac{1}{N^2}\cdot N\sigma^2=\frac{\sigma^2}{N}.$$

Take the square root for the human-readable number: $\sigma_{\bar X}=\sigma/\sqrt N$.

• 💡 **the punchline — the $1/\sqrt N$ law** (see the big-lesson box above): **variance drops as $1/N$, std as $1/\sqrt N$.** This is the chapter's whole quantitative content. Its sting is the diminishing return: $4\times$ the frames for $1\times$ less noise (one stop); 100 frames buys a $10\times$ cleaner image, not 100×. In SNR terms, $+10\log_{10}N$ dB — about **3 dB per doubling**. [`fig-sqrtN-curve`]

• **estimating the noise itself** (aside, for the analysis pipeline): from the same stack you can *measure* $\sigma^2$ per pixel — but divide by $N-1$, not $N$ (**Bessel's correction**), because you used the same samples to estimate the mean, which correlates the residuals and otherwise *under*-estimates the variance. The coin-flip-with-$N{=}2$ example makes this concrete: the $N$ estimator averages to half the true variance; the $N{-}1$ estimator is unbiased.

• **what it confirms about the sensor** (callback to the noise part): map the measured $\sigma$ across a real scene and it tracks the noise sources — more noise (numerically) in **bright** areas (shot noise $\propto\sqrt{\#\text{photons}}$), yet **better SNR** there; the log-SNR image "looks like the photo." Averaging lowers the floor uniformly; it does not change *which* source dominates where.

• **the failure of the mean**: the derivation assumed *clean* IID noise. A real stack has **outliers** — a satellite or plane streaks one frame, a cosmic ray spikes a pixel, someone walks through. The mean is **not robust**: a single wild value drags the average, leaving a faint **ghost trail** exactly where the outlier was. [`fig-mean-vs-median-stack`]

• **median**: take the per-pixel **median** across the stack instead of the mean. One streaked frame is just an extreme value the median ignores → the streak vanishes. Cost: the median is **noisier than the mean** for clean Gaussian data (its variance is $\approx\!1.57\times$ larger, so you lose roughly a third of your $\sqrt N$ benefit) — robustness isn't free.

• **sigma-clipping** (the practical compromise, the astro/stacking default): iterate — compute per-pixel mean and std, **reject** samples more than $k\sigma$ (say $2$–$3\sigma$) from the mean, recompute on the survivors. You get the mean's **efficiency** on the inliers *and* the median's **rejection** of outliers. This is what RegiStax / DeepSkyStacker do under "kappa-sigma" or "winsorized" stacking.

• **the design choice to surface**: this is the same *reject-or-average* decision as edge-preserving filtering (**L4** — use a difference to decide who belongs together), now applied **across frames** instead of across space: a sample too far from its peers is treated as not-belonging. Plain mean = trust everyone; median/sigma-clip = trust the consensus.

• **forward-ref**: the same robust-combine logic reappears as **de-ghosting** in HDR merging and panoramas (reject pixels that disagree because something *moved*), and as the **robust merge** in burst SR ([[Super-resolution and image priors]]).

• **the limit of averaging** (callback to the model): the $1/\sqrt N$ law only kills **zero-mean, independent** noise. Two things break that assumption and survive any amount of averaging — **fixed-pattern** offsets (the *same* error every frame: hot pixels, amp glow, per-pixel dark current, vignetting, dust shadows) and **clipping** (a non-linearity that biases the mean). These need to be *removed*, not averaged. [`fig-calibration-triad`]

• **the calibration triad** — each frame is the inverse of one degradation:

• **bias / offset**: shoot the shortest possible exposure with the lens capped → captures the read-out **electronic offset** (and the deliberate offset many cameras add so noise stays *zero-mean* — see clipping below). **Subtract** it.

• **darks**: shoot capped frames at the **same exposure time and temperature** as your lights → captures **dark current** (thermal charge that accumulates per second) and **hot pixels**. **Subtract** them. *This is why astro cameras are **cooled and temperature-regulated*** — dark current roughly doubles every 6–8 °C, and a *repeatable* temperature makes the dark frame actually match the light frame (callback to FUNDAMENTALS — cooled cameras, **L15** lower floor).

• **flats**: shoot an evenly-lit blank field → captures the **multiplicative** response variation — **vignetting** (corners darker) and **dust** shadows. **Divide** by it (normalised).

• in one expression: $X_\text{cal}=\dfrac{X_\text{light}-X_\text{dark}}{X_\text{flat}-X_\text{bias}}$ — subtract the **additive** fixed-pattern terms, divide out the **multiplicative** ones. (And you average *many* of each calibration frame too, so they don't re-inject noise — $1/\sqrt N$ all the way down.)

• **the clipping bias** (a subtle one worth a figure): the sensor **clamps at 0 and at max**, so in deep shadows the noise is *not* zero-mean — the below-zero excursions are cut off, leaving only the positive ones. Averaging then pulls a true-black region **too bright**. The fix is a **deliberate positive offset** (Canon's trick) added before read-out so the full noise distribution stays above 0 and *stays zero-mean*; subtract it back in calibration. [`fig-clipping-bias`]

• **dithering** (the trick that *converts* fixed-pattern into averageable noise): deliberately **nudge the framing** a pixel or two between frames. After re-alignment, any residual fixed-pattern error lands on a *different* part of the sky each frame, so it becomes effectively random across the registered stack — and *then* averaging/clipping can remove it. (This is also why burst SR's hand-tremor is a feature, not a bug — [[Super-resolution and image priors]].)

• the purest use of "average to denoise": a faint nebula/galaxy buried in noise and light pollution → shoot **many long "subs"**, **align** them (the sky rotates — cross-ref [[#Image alignment]] below), and **average** → signal stays while noise falls as **1/√N**, pulling structure out of near-nothing. This is the $1/\sqrt N$ law pushed to its physical limit: hours of total integration (hundreds of subs) to lift a galaxy that is *invisible* in any single frame. [`fig-astro-stack-reveal`]

• **calibration frames** matter as much as the lights: subtract **darks** (dark current + hot pixels — why cooled astro cameras regulate temperature), divide by **flats** (vignetting + dust), remove **bias/offset** — the fixed-pattern terms of the noise chapter, taken out by calibration (see *Calibration frames* above — astro is where the triad is non-negotiable).

• **robust combine, not a plain mean**: per-pixel **sigma-clipping / median** across the stack rejects satellites, planes, and cosmic rays; **dithering** (nudging the framing between subs) turns fixed-pattern residue into something averaging *can* remove (see *robust combination* and *dithering* above). [`fig-mean-vs-median-stack`]

• the budget: read noise per sub favors **fewer, longer** subs; saturation, tracking error, and clipping favor **more, shorter** ones — the same exposure trade-off, astro edition (cross-ref [[Noise, signal-to-noise ratio and dynamic range]], ETTR). Quantitatively: once each sub's signal is well **above the read-noise floor**, lengthening subs no longer helps the $\sqrt N$ math — but it reduces the *number* of read-noise events, so when read noise is significant (light-polluted or narrowband), **fewer-longer** wins; when tracking is shaky or stars saturate, **more-shorter** wins. [`fig-sub-length-tradeoff`]

• cross-ref: **cooled cameras** (FUNDAMENTALS *Cameras beyond photography*); **lucky imaging** for planetary (Big data — pick the *sharpest* frames rather than averaging all, when atmospheric seeing dominates); the **astro hub** (Adjacent fields). Note the contrast with planetary lucky imaging: deep-sky *averages everything* to beat read/shot noise; planetary *selects the best few percent* to beat atmospheric blur — same burst, opposite combine rule.

• **the precondition**, made vivid: take a hand-held burst and average it *without* aligning → you don't get a cleaner image, you get a **blurry double**. The $1/\sqrt N$ benefit only materialises once the frames agree pixel-for-pixel; alignment is what *earns* it. [`fig-align-before-average`]

• **the two failure scales**: a **sub-pixel** misalignment loses fine detail (the average smears across the offset — a soft low-pass); a **whole-pixel-or-more** misalignment **ghosts** (edges appear twice). In astro the driver is unavoidable — the **sky rotates** through the exposure, so without per-sub alignment the stars trail and the stack is mush.

• **the reframe**: alignment is itself an **inverse problem / optimization** — find the transform $T$ that makes $I_2(T\!\cdot)$ match $I_1$, i.e. $\hat T=\arg\min_T\sum_x\big(I_1(x)-I_2(Tx)\big)^2$. Everything below is *which family of $T$* (translation, homography, dense flow) and *how you search it*.

• **the simplest model — pure translation**: for a tight burst, frame-to-frame motion is well-approximated by a 2-D **shift** $(dx,dy)$. Try **all shifts** within $\pm\,$maxOffset and keep the best-scoring one. (This is literally a pset: brute force over a shift window.)

• **the score — SSD**: sum of squared differences, $\mathrm{SSD}(dx,dy)=\sum_{x,y}\big(I_1(x,y)-I_2(x{+}dx,y{+}dy)\big)^2$ — small when the frames agree. Plot it over the shift window and you get a **basin with a clear minimum** at the true shift. [`fig-ssd-search-surface`]

• *pseudocode*: the brute-force aligner as a double loop over the shift window — score each candidate by SSD, keep the argmin.

• **when SSD lies — NCC**: SSD assumes **brightness constancy** (same exposure/illumination). Across an HDR bracket or with a brightness drift, that's false and SSD breaks. **Normalised cross-correlation** is invariant to a per-frame **gain and offset** — subtract each patch's mean, divide by its std, correlate: $\mathrm{NCC}=\dfrac{\sum (I_1-\bar I_1)(I_2-\bar I_2)}{\sqrt{\sum(I_1-\bar I_1)^2}\,\sqrt{\sum(I_2-\bar I_2)^2}}$, **maximised** at the true alignment. Use SSD for same-exposure bursts, NCC (or align in a normalised/gradient domain) when brightness changes between frames.

• **debugging / hand-checkable input**: align a synthetic image to a copy of itself **shifted by 1–2 px** — the alignment score (e.g. SSD of the difference) must be **smallest at the true shift**; validate the search this way before trusting real data. Concretely: shift a known image by $(+2,0)$, run the search, and confirm the score map's minimum sits *exactly* at $(+2,0)$. If it doesn't, your indexing/sign/interpolation is wrong — and you'll never catch it on noisy real frames. This shifted-copy test is the cheapest, most reliable unit test an aligner has. [`fig-shifted-copy-debug`]

• **sub-pixel refinement** (forward-ref): integer-shift search gets you close; fit a parabola to the SSD basin (or upsample) for **sub-pixel** accuracy — which is exactly the precision burst **super-resolution** depends on ([[Super-resolution and image priors]]). Shifting by a fractional pixel means **interpolation** (bilinear/bicubic — BASIC Resampling).

• **the idea** (and the lesson): a **pure shift** is, in Fourier, just a **phase ramp** — the magnitudes are unchanged, only the phases tilt: $I_2(x)=I_1(x-d)\Leftrightarrow\hat I_2(\omega)=\hat I_1(\omega)e^{-i\omega\cdot d}$ (the **shift theorem**). So the *entire* translation lives in the phase. 💡 **Big lesson (L5 · diagonalize when you can):** convolution and correlation are **diagonal in Fourier** — instead of an $O(\text{window}^2)$ search over shifts, one FFT pair recovers the global shift directly. (→ see Big lesson **L5**, first placed in BASIC — [[Linearity, Fourier, Aliasing and deblurring]]; here it turns brute-force search into a single transform.)

• **the algorithm** — phase correlation (Kuglin & Hines 1975): form the **normalised cross-power spectrum** $R(\omega)=\dfrac{\hat I_1\,\overline{\hat I_2}}{|\hat I_1\,\overline{\hat I_2}|}=e^{i\omega\cdot d}$ (normalising to *unit magnitude* keeps only the phase — robust to brightness and to dominant low-frequency content). Its inverse transform $\mathcal F^{-1}\{R\}$ is a **single sharp delta at the shift $d$** — read off the peak location. [`fig-phase-correlation-spike`]

• **why it's nice**: global, fast ($O(n\log n)$), and the delta's **sharpness/height** doubles as a confidence measure. Extends (log-polar — Reddy & Chatterji 1996) to recover **rotation and scale** as shifts in a transformed domain.

• **its limits** (be honest): it assumes a **single global shift** over the whole frame — it can't handle parallax, scene motion, or a per-region warp. Those need a richer transform (homography → panorama chapter) or **per-pixel** motion (optical flow → video).

• **the two problems brute force has**: large motions blow up the search window (cost), and a raw SSD surface can have **local minima** (texture/aliasing) that trap a naive search.

• **the fix — multiscale**: build an **image pyramid** (cross-ref [[Linear pyramids and wavelets]]) and align **coarse-to-fine**. At the coarsest (tiny, blurred) level a *small* shift covers a *large* real motion and the score surface is smooth (no local minima) → get a cheap rough estimate; **propagate it down**, refining at each finer level with only a tiny local search. [`fig-coarse-to-fine-pyramid`]

• **why it works** (frequency view, **L5/L16**): the coarse level keeps only **low frequencies**, whose SSD basin is wide and unimodal; finer levels add the high-frequency detail that sharpens the estimate but would otherwise create false minima. Prefiltering on the way down the pyramid is the same **L16** "blur before you decimate" discipline.

• **in production**: HDR+ (Hasinoff 2016) aligns burst frames this way — a **tiled, coarse-to-fine** search per tile, then a robust merge — handling both global hand motion and local scene motion at once.

• **the hand-off to video**: replace "one transform per frame" with "**one motion vector per pixel**" and coarse-to-fine matching becomes **optical flow**; constrain $T$ to a homography and it becomes **stabilization / panorama**. The alignment machinery here is the entry point to [[Slopendium/09 Motion, warping, and video|Motion, warping, and video]] — *this chapter links forward there.*

• **the gap, stated in numbers**: real scenes span **10 to 12 orders of magnitude** of illumination (the eye adapts from $\sim 10^{-6}$ to $10^{6}$ cd/m²; a single indoor-with-window scene is routinely $1{:}100{,}000$). A negative or sensor records only **2–3 orders** in one shot; a print or display shows even less — typically $1{:}20$ to $1{:}50$, at best $\sim 1{:}500$. *Two distinct problems fall out:* **(1) record** the information (a single capture can't), and **(2) display** it (the medium can't show it even once recorded). This chapter is problem (1) — **recording** via merge. Problem (2), **tone reproduction**, is forward/elsewhere. [`fig-dr-clip-vs-noise`]

• **why one exposure fails — the two walls** (this *is* **L15**): turn exposure up and the **highlights clip** — any radiance past the photosite's full-well capacity saturates to the max value, all detail above it collapses to white. Turn exposure down and the **shadows drown** — the signal sinks below the **read-noise floor**, and brightening them in post just amplifies noise. The window between the two walls is the **single-shot dynamic range**; a high-DR scene doesn't fit. [`fig-dr-clip-vs-noise`]

• **the formation model** (carry it through the whole chapter): scene radiance $L(x,y)$ reaches a photosite, is scaled by an **exposure factor** $k_i$ (set by shutter, aperture, ISO), has **noise** added, and **clips** above the well capacity: $I_i(x,y)=\mathrm{clip}\big(k_i\,L(x,y)+n\big)$. Everything below is *invert this*: estimate the single $L$ from the several clipped, noisy $I_i$.

• **the resolution — bracket and merge** (this *is* **L14**): shoot $N$ exposures that **sequentially measure different segments of the range** — a short one keeps the highlights, a long one digs out the shadows. Each $I_i$ is reliable only over a *band* of radiance (above its noise, below its clip); slide the band with $t_i$ and the bands tile the whole scene range. Merge the reliable parts into one floating-point radiance map. [`fig-exposure-stack-slices`]

• **scope note (what this chapter is *not*)**: the output is a **linear radiance map** — one float per $(x,y,c)$, values may be $\gg 1$ (stored in OpenEXR / RGBE / DNG; see [[#Application to cell phones: HDR+ and burst imaging]] for the raw-domain variant). **Tone mapping** — compressing that map to a displayable image while preserving local detail (bilateral / gradient-domain operators) — is a separate stage, cross-referenced to [[Edge-preserving techniques]] and recapped in [[Single-image computational photography]]. *Merge here, tone-reproduce there.*

• **caveat — modern single-shot DR**: today's sensors already give $\sim 13$–14 stops in one capture, so HDR-merge is less universally necessary than it was; even a single raw can benefit from *tone mapping* alone (noise in the shadows is the limit). HDR merge still wins when the scene exceeds the sensor's single-shot range — and, crucially, the **cell-phone burst** (next chapter) reuses the same merge for *denoising*, not only range.

• **the old, analog precursors** (history, ties to the chapter's opening): photographers fought dynamic range long before digital — Gustave Le Gray (1857) **sandwiched two negatives** (one exposed for sky, one for sea); fill-flash and **graduated ND filters** compress scene contrast at capture; the darkroom's **dodging and burning** and **Zone System** are tone reproduction by hand. Surface these as the lineage; the digital merge automates the negative-sandwich.

• **the question**: to tile the radiance axis you need to *change the exposure factor* $k_i$ between shots. Four knobs do it — **shutter speed, aperture, ISO, neutral-density (ND) filter** — and they are **not** interchangeable: each has a side effect on the image beyond brightness. [`fig-vary-exposure-knobs`]

• **shutter speed — the clean knob (use this)**: range $\sim$30 s to 1/4000 s ≈ **6 orders of magnitude**. **Pros: reliable and linear** — doubling exposure time doubles collected photons, so $k_i\propto t_i$ exactly, which is what the merge math assumes. **Con:** very long exposures add their own noise (dark current) and risk motion. This is the default for bracketing because it changes *only* the exposure, nothing else about the image.

• **aperture — changes the picture**: range $\sim f/1.4$ to $f/22 \approx$ **2.5 orders** ($k\propto 1/N_f^2$). **Con:** it changes **depth of field** — the bracket frames would have *different focus blur*, breaking the "only the exposure changes" assumption. Use only when desperate.

• **ISO — amplifies, doesn't collect**: range $\sim$100 to 1600 ≈ **1.5 orders**. ISO is **analog/digital gain applied after capture**, not more light — so it scales signal *and* read-noise-referred terms and generally **raises noise**. Useful when desperate, but a poor way to extend range on its own. (Hasinoff's insight, below, is that ISO *is* a useful degree of freedom once you model noise properly.)

• **neutral-density (ND) filter — darkens the world**: up to $\sim$4 densities ≈ 4 orders, **stackable**. **Cons:** not perfectly neutral (introduces a **color shift**), imprecise, and you must **touch the camera** to add it (risking shake → needs re-registration). **Pro:** works with strobe/flash where shutter can't, a good complement when desperate.

• **the takeaway** (ties to **L7** — stops): exposure is **counted in stops** (factors of 2, additive in log space), and the clean way to walk the radiance axis is **shutter**; the other knobs trade a side effect (DoF, noise, color) for range and are fallbacks. *Forward-ref:* which exposures to pick — not just *how* to vary, but the **optimal set** — is the *Optimizing the capture and merge* section (Hasinoff).

• **the merge assumes static + registered**: the basic merge assumes **only the exposure changes — no motion**. Hand-held bracketing breaks this; align first (Ward 2003's **median-threshold bitmap**, MTB — a contrast-robust alignment that compares images thresholded at their median, accelerated on a pyramid) and reject **ghosts** from moving objects. Cross-ref [[#Image alignment]]; the burst-imaging chapter makes robust alignment central.

• **on-chip HDR — the sensor does the merge (no software bracket)**: everything above assumes *software* fuses a bracket of separate frames. But **on-chip HDR is now an industry-wide sensor capability** — many modern image sensors **produce a high-dynamic-range signal directly, on-chip**, folding the "capture several slices and combine" step into the readout itself. **Sony** is a prominent example (its DOL / dual-read schemes, below), but **OmniVision**, **Samsung (ISOCELL)**, and **onsemi** and other automotive vendors (split-pixel designs) all do on-chip HDR by various strategies — it's a hardware strategy, not one vendor's feature. This is the hardware cousin of bracketing: instead of $N$ frames merged in post, the slices are captured and combined *inside the sensor / ISP*, so the photographer gets one wide-DR raw out. The main strategies differ in **how** they multiplex the dynamic range — in **time**, in **space**, or in **gain**: [`fig-hdr-sensor-strategies`]

• **DOL-HDR / "double read" (digital overlap)** — read the same exposure twice, or interleave **long + short** exposures line-by-line, then combine; the long read recovers shadows, the short read the highlights. This is the Sony **"double read."** *Time-multiplexed:* the long and short captures are not simultaneous, so anything moving between them **ghosts** (motion artifacts) — the same de-ghosting problem as a hand-held bracket, now inside one frame.

• **dual-conversion-gain (DCG)** — each pixel can be read at **two readout (conversion) gains**: high gain for a low noise floor in the shadows, low gain for more full-well headroom in the highlights; the two reads are merged. *Gain-multiplexed:* both reads see the **same instant**, so it is the **cleanest** option — no motion artifacts, no resolution loss — but the extra range it buys is **bounded** by the gain spread.

• **split-pixel / dual-photodiode** — each photosite carries a **large + small** sub-photodiode: the large one saturates early (shadows/midtones), the small one keeps reading into bright highlights. Common in **automotive** sensors (sun-glare, tunnel exits). *Space-multiplexed:* costs **resolution / fill-factor and SNR** (the small diode is noisy), but both sub-pixels are exposed simultaneously, so no motion ghosting.

• **lateral-overflow / LOFIC / voltage-domain tricks** — add **extra charge capacity** per pixel (a lateral-overflow integration capacitor that catches the charge spilling past the photodiode's well), extending full-well in the **voltage/charge domain** without a second exposure.

• **spatially-varying exposure (SVE / "assorted pixels")** — bake **different sensitivities into neighbouring pixels** (an exposure mosaic, like a Bayer pattern but for ND), then demosaic into an HDR image — **Nayar & Mitsunaga**. *Space-multiplexed:* trades **resolution** for range, single-shot so no motion artifacts.

• **logarithmic / companding sensors** — a pixel whose response is **log (or piecewise-companded)** in radiance, capturing many decades in one read at the cost of **lower contrast resolution / fixed-pattern non-uniformity** in any given band.

• **the trade-off, in one line**: **time-multiplexed** schemes (DOL double-read) buy range with **motion artifacts**; **space-multiplexed** schemes (split-pixel, SVE/assorted pixels) buy it with **resolution / SNR**; **gain-domain** (DCG) is the **cleanest** — same instant, full resolution — but its extra range is **bounded**. The software **HDR+ burst** of the next chapter ([[#Application to cell phones: HDR+ and burst imaging]]) sits at the opposite end: maximal flexibility and denoising-for-free, paid for in compute and alignment. This is the **computational-sensor** trade space — push the work into silicon or into software — developed in [[Advanced computational photography]] (cross-ref). [`fig-hdr-sensor-strategies`]

• **the calibration question**: to merge, you must know how a pixel value $z_i$ relates to scene radiance — i.e. recover the **exposure factors** $k_i$ (or their ratios) and, if the capture is non-linear, the **camera response curve** $f$ where $z = f(k\,L)$. Two regimes: **linear capture** (easy, a ratio) and **non-linear capture** (regression on pixels seen at multiple exposures).

• **linear case — just a ratio**: if the images are **linearly encoded** (raw, or un-gamma'd — `dcraw -g 2.2 0` gives linear-ish; **L2**: do radiometry in linear light), then in any non-clipped, non-noisy pixel $I_i = k_i L$. So for a pair of adjacent exposures the radiance cancels: $I_i/I_j = k_i/k_j$ — *the same value at every good pixel* if linearity holds. Estimate it robustly: $k_i/k_j=\mathrm{median}_{x:\,w_i,w_j>0}\big(I_i(x,y)/I_j(x,y)\big)$, taking the **median** over pixels where **both** images are reliable. **Chain** the pairwise ratios to get every $k_i/k_0$ — you only ever recover exposure up to **one global scale factor** (which is fine: radiance maps are relative unless you photometrically calibrate). [`fig-pixel-ratio-calibration`]

• *practical note:* you can also just read $k_i$ from the **EXIF exposure data** ($t_i$, $f$, ISO); the ratio-from-pixels method is the fallback when the metadata is missing or the response is suspect, and it's more robust because it's measured from the actual data.

• **non-linear case — recover the response curve (Debevec–Malik 1997)**: real cameras (and film, and JPEG) apply a **non-linear response** $f$ (gamma, tone curve, film characteristic). Then $z_i = f(k_i L)$ and you must recover $f$ (equivalently its log-inverse $g=\ln f^{-1}$) before you can merge. The trick: a pixel imaged at **several exposures** gives several $(z_i, t_i)$ pairs constraining the *same* radiance — that over-determination pins down the curve. Write, in the log domain, $g(z_i) = \ln L + \ln t_i$. The unknowns are the curve $g(\cdot)$ (one value per possible pixel level, e.g. 256) and one $\ln L$ per pixel. Solve a **linear least squares** over all pixels and exposures, with two riders: a **smoothness penalty** on $g''$ (a real response is smooth), and a **reliability weight** $w(z)$ down-weighting near-clip and near-noise samples. Recovered only up to **one additive constant** (= the global scale). [`fig-response-curve-debevec`]

• **Mitsunaga–Nayar 1999 — even less is assumed**: models the response as a **low-order polynomial** and recovers it (and the unknown exposure ratios jointly) by **radiometric self-calibration** — you don't even need to trust the EXIF exposure ratios. Same spirit: pixels seen at multiple exposures over-determine the curve; fit a smooth response. (Mann & Picard 1995's "comparametric" / Wyckoff work is the earlier multiple-exposure precursor.)

• **why regression beats raw ratios** (the cleaner-way bridge): a **robust regression** for $g$ and $L$ avoids brittle per-pair ratios, **directly models uncertainty due to noise** (heavier weight where SNR is high), and **generalizes to the full non-linear response** in one framework. This is the seam into *Optimizing the merge*: once you've written it as least squares, the *right* weights are inverse-variance — the next two sections.

• **encoding note (L2)**: merging is an **additive** operation in radiance, so it must be done in **linear light** (or, equivalently, the log-radiance domain of Debevec). Gamma/tone-curved values are *not* proportional to radiance and would merge wrong — un-apply the response first. (The *display* side, by contrast, lives in a perceptual/log space — that's tone mapping, elsewhere.)

• **the assembly recipe** (three steps): (1) figure out the **scale factor** $k_i$ between images — from EXIF or pixel ratios (previous section); (2) compute a **weight map** $w_i(x,y)$ for each image marking which pixels are usable; (3) **reconstruct** the radiance per pixel as a **weighted combination** of the rescaled observations. [`fig-weight-triangle`]

• **which pixels are useful — the reliability mask**: at each pixel, an exposure's value is trustworthy only if it is **neither clipped nor noise-dominated**. Reject **clipped** highlights (e.g. $z > 0.99$) — they only tell you "at least this bright," not how bright. Reject **too-dark / too-noisy** shadows (e.g. $z < 0.002$) — the read-noise floor swamps the signal. What's left is the usable band. (P-set 5 uses **binary** $w\in\{0,1\}$, extensible to a smooth scalar; weights can differ per color channel.)

• **the merge — a weighted average in radiance**: rescale each usable observation to radiance by dividing out its exposure factor, $z_i/t_i$ (or $z_i/k_i$), then average, weighted by reliability:

$$\hat L(x,y)=\frac{\sum_i w(z_i)\,z_i/t_i}{\sum_i w(z_i)}.$$

In the non-linear (Debevec) case, the identical average runs in the **log domain** on the recovered curve: $\ln\hat L=\big(\sum_i w(z_i)\,(g(z_i)-\ln t_i)\big)\big/\big(\sum_i w(z_i)\big)$. Either way it is a **reliability-weighted mean** — and where several exposures overlap, the averaging also **denoises** (the $1/\sqrt N$ effect of [[#Denoising by averaging]], reliability-weighted). [`fig-weight-triangle`]

• *pseudocode*: the merge as one per-pixel pass over the bracket — divide out each exposure factor, accumulate by reliability weight, normalize, with the all-bad fallback to the brightest/shortest frame.

• **edge case — pixels bad in *every* exposure**: a pixel may be **clipped in all** images (a light source) or **noise-dominated in all** (deepest shadow). Don't discard it: **keep clipped pixels in the *brightest*-exposed image** (the one shot long enough to have *some* value there — for the darkest part) and **keep dark pixels in the *shortest* exposure**. Careful with the threshold implementation: the literal values $0.0$ / $1.0$ must actually be *kept* for these fallback images, not rejected. (Mind the direction: dark pixels of the *brightest* image, bright pixels of the *darkest* — not the reverse.)

• **the connection to denoising and to fusion** (the unifying view): the unweighted, single-radiance-band case of this average is exactly **average-to-denoise** ([[#Denoising by averaging]]) — HDR merge generalizes it with per-observation reliability/SNR weights. A *different* philosophy is **exposure fusion** (Mertens, Kautz & Van Reeth 2007): skip the radiance map entirely — score each bracket pixel by a **per-pixel quality** (well-exposedness, local contrast, saturation), then **blend the bracket directly in a Laplacian pyramid** weighted by those scores. No response-curve recovery, no HDR float map, no separate tone-mapping step — the multi-scale (pyramid) blend hides the exposure seams and yields a *displayable* image in one shot. Cross-ref the pyramid-blend machinery in [[Linear pyramids and wavelets]]; fusion is the pragmatic "merge straight to a viewable image" alternative to "build an HDR map, then tone-map."

• **the better question** (beyond binary weights): when **multiple exposures both see a pixel validly**, they have **different noise**. Plain binary masks throw away information — using *only* the least-noisy observation gets no $1/\sqrt N$ benefit; there must be a way to use the noisier one *at least a little*. The principled answer: weight by **reliability = inverse noise variance**.

• **the two-observation derivation** (the kernel of it): given two unbiased estimates $x,y$ of the same quantity with variances $\sigma_x^2,\sigma_y^2$, combine as $\hat L = a x + (1-a) y$. Its variance is $a^2\sigma_x^2 + (1-a)^2\sigma_y^2$; differentiate w.r.t. $a$, set to zero → $a^\* = \dfrac{\sigma_y^2}{\sigma_x^2+\sigma_y^2}$. The **optimal combination weights each estimate by the inverse of its variance** (check: equal variances → $a=1/2$, the plain average; one estimate far noisier → it gets little weight but not zero). Generalize: $w_i = 1/\sigma_i^2$, giving $\hat L = \dfrac{\sum_i (z_i/k_i)/\sigma_i^2}{\sum_i 1/\sigma_i^2}$ — **inverse-variance (minimum-variance) weighting**, the BLUE estimator. [`fig-naive-vs-optimal-weights`]

• **plug in the camera's noise model**: variance is dominated by two terms (FUNDAMENTALS — Noise): **photon (shot) noise** with variance $\propto$ signal (dominates in the highlights) and **read noise** with constant variance (dominates in the shadows): $\sigma^2(x) = a\,x + \sigma_{\text{read}}^2$, where $a$ and $\sigma_{\text{read}}$ depend on camera and ISO. Calibrate by shooting repeated frames (variance across the stack) or derive it on paper. Now the merge weights aren't a hand-drawn hat — they come from physics.

• **what falls out** (a clean, slightly surprising result): rescaling an observation by $1/k_i$ to radiance amplifies **both signal and noise**, so $\mathrm{Var}(z_i/k_i)\approx \sigma_i^2/k_i^2$. Working the algebra through the inverse-variance formula for the **photon-noise-only** case: the per-pixel radiance $L$ and the slope $a$ are common to every term (they appear in numerator and normalizer) and **cancel**, and the two $k_i$ factors cancel too — so **all pixels of a given exposure get the same weight**, $w'_i \propto k_i$ (longer/brighter exposures, with relatively less photon noise, weighted up). The images aren't literally rescaled to radiance in the sum, but the normalization makes it equivalent. (With **read noise** included the cancellation is incomplete and the optimal weights *do* vary per pixel — read noise is what makes the optimum genuinely spatially varying.) Still reject clipped/dark pixels with the binary mask, *then* weight the survivors by inverse variance. [`fig-naive-vs-optimal-weights`]

• **Hasinoff–Durand–Freeman 2010 — optimizing the *capture set*, not just the merge**: don't stop at optimal *weights* for a *given* bracket — choose the **optimal set of exposures** to shoot. With a full noise model (photon + read, and **exploiting ISO** as a real degree of freedom), pick the shutter/ISO sequence that **maximizes worst-case SNR** across the scene's brightness range under a fixed time budget. The counter-intuitive result: the SNR-optimal set often uses **higher ISO and shorter exposures** than naive bracketing — e.g. for one scene, standard bracketing ISO 100 (1/100, 1/25, 1/6 s) is beaten by an SNR-optimal ISO 3200 (1/3200, 1/125, 1/5 s), gaining $\sim$12 dB in the dark regions and approaching the **ideal HDR sensor** curve. [`fig-hasinoff-snr-optimal-set`]

• **the through-line into cell phones**: Hasinoff's "fewer/shorter/higher-ISO, then merge optimally" logic — *underexpose to avoid clipping and to freeze motion, recover the rest by computation* — is exactly the philosophy of **HDR+** (next section), where the bracket becomes a **burst of identical underexposed raws** merged for denoising *and* range at once. (Other refs in this vein: Granados et al. 2010, optimal per-pixel weights from the full noise model; Ward 2003, robust capture with registration + ghost/flare removal.)

• **the phone's predicament**: a tiny sensor (small photosites → low full-well capacity → low single-shot DR and high relative noise — **L15**), **no tripod**, an unsteady hand, and a scene with people moving. Classic bracketing — a few *different* exposures, assume static — fails: the long exposure blurs from hand shake, subjects move between frames (ghosts), and the small sensor's shadows are noisy anyway. [`fig-burst-vs-bracket`]

• **the HDR+ move (Hasinoff 2016)**: replace the bracket with a **burst of many *identical* frames**, each at the **same short, deliberately under-exposed** raw exposure. Short exposures **freeze motion** (no per-frame blur) and are easy to align; *many* of them, **averaged**, beat the noise down by $1/\sqrt N$ — recovering the shadow detail you'd otherwise have gotten from a long exposure, but without the blur. The burst does double duty: **denoising** (the $1/\sqrt N$ of [[#Denoising by averaging]]) **and HDR** (range recovered from the merge) in **one** operation. This is **L14** on a phone — capture the full set (the burst), decide later (merge + tone-map). [`fig-burst-vs-bracket`]

• **why *under*expose — the irreversibility asymmetry** (the crux): the two failure walls are **not** symmetric. A **clipped highlight is gone forever** — once a photosite saturates, the information is destroyed and no merge recovers it. A **dark-but-unclipped shadow is only noisy**, and noise is **recoverable** by averaging $N$ frames. So bias the exposure **down**: pick $k$ small enough that even the **brightest** scene radiance stays below clipping ($k\,L_{\max}<1$), guaranteeing **highlight headroom**, and pay for it with extra shadow noise that the burst merge then removes. You trade a *recoverable* problem (noise) for avoiding an *unrecoverable* one (clipping). [`fig-why-underexpose`]

• **this is Hasinoff 2010 on a phone**: it's exactly the **SNR-optimal capture set** logic — shorter exposures, higher effective ISO, recover by computation — instantiated as "one short exposure repeated $N$ times." The 2010 paper said *how to choose the optimal set*; HDR+ says *for a phone, the optimal set is a uniform under-exposed burst*, because that also solves motion and alignment.

• **the pipeline** (end to end): **capture** a burst of $N$ under-exposed raw frames (raw = linear, pre-demosaic, **L2**) → **select a reference** (a sharp, well-exposed-enough frame, often an early one) → **align** every other frame to the reference (tile-based, sub-pixel) → **robustly merge** the aligned frames **in the raw domain** → emit **one denoised, high-DR raw** → **demosaic**, then **tone-map** and finish (the displayable JPEG). The key architectural choice: **merge happens in raw, before demosaicking**, so denoise + HDR are a single operation on the cleanest (linear, mosaiced) data. [`fig-hdrplus-pipeline`]

• **the merge is a robust weighted average** — the [[#HDR merging]] merge with one addition: $\hat L = \sum_i w_i\,I_i(W_i^{-1}x)/\sum_i w_i$, where $W_i$ is the estimated alignment of frame $i$ and the weight $w_i$ combines **inverse-variance reliability** (from the noise model) with **outlier rejection** — a frame (or tile) that *disagrees* with the reference (because a subject moved, or alignment failed) is **down-weighted or dropped**, so it can't contribute a ghost. Conceptually a per-frequency **Wiener-style merge** in the tile transform: where frames agree, average hard (full denoise); where they disagree, trust the reference (no ghost). [`fig-ghost-from-misalignment`]

• **alignment is the whole game** (and ghosts are its failures): mis-registration turns the helpful average into **blur and double-images**. HDR+ aligns in **tiles** (a coarse-to-fine, pyramid search per tile) to handle locally-varying motion, and the **robust** merge is the safety net for the residual motion the alignment can't undo (occlusion, fast subjects). The trade is explicit: more aggressive averaging = more denoising but more ghost risk; the robustness weight tunes it per pixel. (Cross-ref [[#Image alignment]]; this is the same MTB/pyramid-search lineage as Ward 2003, made local and sub-pixel.)

• **why raw, before demosaic**: doing the merge on **linear raw** keeps the noise model simple and correct (photon + read, **L15**), lets the merge *also* serve as the demosaic's denoiser, and avoids baking in the non-linear tone curve before the additive average (**L2** — average in linear light). Tone mapping comes **after** the merge (local tone mapping; HDRnet / Gharbi 2017 learns this step on a **bilateral grid** — forward-ref to [[Edge-preserving techniques]] and [[Deep learning]]).

• **the same substrate, one knob turned**: HDR+ aligns and merges a burst to gain **SNR and range**. **Burst super-resolution** (Wronski et al. 2019, the Pixel "Super Res Zoom") aligns and merges the *same kind of burst* to gain **resolution**. The only difference that matters is **sub-pixel** alignment: the unavoidable **hand tremor** between frames shifts the scene by fractions of a pixel, so each frame samples the continuous scene on a slightly **offset grid** — fuse them on a **finer grid** and you recover detail beyond a single frame's sampling. [`fig-burst-superres-link`]

• **aliasing becomes the signal**: a single frame's sampling **folds** high frequencies (aliasing — normally a defect); the sub-pixel-shifted frames let you *separate* those folded frequencies, turning aliasing into recovered resolution. This is **reconstruction-based** super-resolution (genuine measured detail), not hallucination — cross-ref the reconstruction-vs-hallucination axis in [[Single-image computational photography#Reconstruction vs hallucination]]. It even **replaces demosaicking**: the color mosaic's own per-channel offsets, combined across the burst, reconstruct full color directly. [`fig-burst-superres-link`]

• **the unifying statement** (close the chapter): HDR merge, burst denoising, and burst super-resolution are **one operation** — *align a burst, then robustly merge* — with the dial set to recover **range** (HDR+), **SNR** ($1/\sqrt N$ denoise), or **resolution** (sub-pixel super-res), and usually **all three at once** on a modern phone. The full super-resolution treatment (the prior, the ill-posedness, the fusion math) lives in [[Single-image computational photography]]; here it closes the multiple-exposure arc — *capture the set, align, merge, decide later* (**L14**), beating the single-shot limits of **L15** by computation.

• **the setup**: you stand in one place and **pan the camera** across a scene, taking overlapping frames. Choose one frame as the **reference**; reproject (warp) the others into its coordinate system; blend the overlaps. The output is a virtual **wide-angle** view. [`fig-pano-4clicks`]

• **the one rule** — pure **rotation about the optical centre**. The camera may rotate freely (pan, tilt, roll) but its **viewpoint must not move**. If you keep the optical centre fixed, every pair of views is related by a single homography (derived below) *regardless of scene geometry*. [`fig-pano-rotate-vs-translate`]

• **why translation breaks it**: if you *walk* (translate the optical centre), near and far objects shift by **different** amounts — **parallax**. There is then no single 2D map between the views; you'd need the depth of every pixel (i.e. actual 3D reconstruction). The slogan for the reader: **"rotate, don't walk."** The one exception is a **planar scene** (next-to-last section) — there a homography works *even if you translate*, because a plane has, in effect, one depth law.

• **forward-pointer**: the *warping itself* — forward vs inverse warp, resampling, antialiasing — and the closely-related **perspective-distortion correction** are developed in [[Slopendium/09 Motion, warping, and video|Motion, warping, and video]]. Here we only need the **geometry** (which $H$) and how to **solve** for it.

• **the pinhole model**: a 3D point $(x,y,z)$ in the camera frame images to $(x/z,\,y/z)$ on the plane $z=1$. **Projection is division by depth.** Far things look small because you divide by a bigger $z$. [`fig-pinhole-divide-by-z`]

• **rotating the camera = rotating the world the other way**: to project a point into a camera that has been rotated by $R$, apply $R^{-1}$ (equivalently $R$ to the ray) then divide. So a rotated view of the *same* scene is: rotate the 3D points, then re-project (divide by the new $z'$).

• **the trap we're about to escape**: this recipe needs the 3D point — and for stitching we only have the **first image's pixels**, not their depths $z$. Are we stuck? (No — that's the next section.)

• **the key observation** — *all points along one viewing ray reproject to the same place.* A pixel $(x,y)$ in image 1 is the image of an entire **ray** of 3D points $(tx,ty,t)$ for every depth $t$. Rotate that whole ray by $R$ and re-project: the **divide by $z'$ removes the common factor $t$**. Every point on the ray lands on the *same* pixel in the rotated view. So the destination pixel **does not depend on depth at all**. [`fig-depth-cancels-ray`]

• **therefore pick $z=1$**: since depth doesn't matter, treat each pixel as the point $(x,y,1)$ — pretend the image is a plane at distance 1 — rotate by $R$, divide by the new third coordinate. That is the entire mapping. We *chose* a depth only for convenience; any choice gives the same answer. [`fig-homog-coords-1d` shows the 1D toy: reprojecting points to another line is linear and depth-free]

• **the equation** — chaining "un-project (apply $K^{-1}$), rotate ($R$), re-project (apply $K$)" gives the view-to-view map

$$\mathbf{x}' \simeq H\,\mathbf{x}, \qquad H = K\,R\,K^{-1},$$

where $K$ is the camera's intrinsics (focal length, principal point) and $\simeq$ means "up to scale" (you still divide by the last coordinate). **Crucially, no $z$ appears in $H$** — depth has cancelled. For normalised coordinates ($K=I$) it's just $H=R$ acting on rays $(x,y,1)$.

• **the punchline to state plainly**: *for a camera rotating about its optical centre, the relationship between two images is a homography that is independent of the scene.* No depth, no 3D model, no parallax — one $3\times3$ matrix relates the views everywhere. This is the chapter's central idea and the reason panorama stitching is a clean linear problem. (Reused in [[#Automatic panorama stitching from multiple views and feature matching|automatic stitching]] and in perspective correction.)

• **homogeneous coordinates** — represent a 2D point $(u,v)$ by **three** numbers $(x,y,w)$ with $u=x/w,\ v=y/w$. Recover the Euclidean point by **dividing by the last coordinate**. All scalings $(wx,wy,w)$ name the same point → coordinates are defined **up to scale**. Motivations: they make **perspective** (the divide) and **translation** expressible as a single matrix multiply, and they're exactly "treat the image as a plane at $z=1$ in 3D." [`fig-homog-coords-1d`]

• **a homography** is any $3\times3$ matrix $H$ acting on homogeneous coordinates: compute $\mathbf{x}'=H\mathbf{x}$ (plain matrix multiply), then **divide by $w'$** to read off image coordinates. It is the most general **projective mapping of a plane** — and, per the previous section, exactly the camera-rotation map. [`fig-homography-quad`]

• **what it does geometrically**: maps the input **rectangle to an arbitrary quadrilateral**; parallel lines need not stay parallel (they can converge to a vanishing point), but **straight lines stay straight**. Same as "project the plane, rotate, re-project." [`fig-homography-quad`]

• **8 degrees of freedom**: $H$ has 9 entries but is defined **up to scale** ($H$ and $kH$ give the same map, since $(kw'x',\dots)$ and $(w'x',\dots)$ are the same Euclidean point) → **8 DOF**. This is why **4 point correspondences** (8 numbers) are exactly enough to pin it down.

• **the per-point formula** (writing $H=\begin{psmallmatrix}a&b&c\\ d&e&f\\ g&h&i\end{psmallmatrix}$): $\ x'=\dfrac{ax+by+c}{gx+hy+i},\quad y'=\dfrac{dx+ey+f}{gx+hy+i}.$ The denominator $w'=gx+hy+i$ is the perspective divide; $g,h$ are what make lines converge (zero $g,h$ → an affine map, no perspective).

• **the goal**: given $\ge4$ matched point pairs $\mathbf{x}_i\leftrightarrow\mathbf{x}_i'$, find $H$ mapping each $\mathbf{x}_i$ to $\mathbf{x}_i'$. In manual stitching the user supplies these by clicking; automatically they come from feature matching (next chapter).

• **make it linear**: the relation $\mathbf{x}'\simeq H\mathbf{x}$ looks nonlinear because of the divide, but **cross-multiplying** by the denominator clears it. Each correspondence yields **two linear equations** in the 9 unknown entries of $H$ — e.g. one of them is $a x + b y + c - x'(g x + h y + i)=0$. Stack them: $A\mathbf{h}=\mathbf{0}$, where $\mathbf{h}$ is the 9-vector of $H$'s entries. With 4 pairs you get $8$ equations.

• **the scale freedom** (and how to kill it): $\mathbf{h}=\mathbf{0}$ is a trivial solution and $H$ is only defined up to scale, so you must **fix the scale**:

• **dirty version** — assume $i\neq0$ and **set $i=1$** (true unless the camera is rotated ~90°). Then 8 unknowns, $A\mathbf{x}=\mathbf{b}$ with a nonzero right side; solve directly (4 pairs) or by **least squares** if you have more pairs (overconstrained $\arg\min\lVert A\mathbf{x}-\mathbf{b}\rVert^2$, the normal equations $A^\top A\mathbf{x}=A^\top\mathbf{b}$). Good enough for a problem set.

• **clean version** — keep all 9 unknowns and solve the homogeneous $A\mathbf{h}=\mathbf{0}$ by **SVD**: $H$ is the **singular vector with the smallest singular value** (the null space). No arbitrary assumption about $i$. [link to [[Linear Inverse Problems and Regression]] for the SVD null-space solve]

• **overdetermined is good**: more than 4 correspondences → solve in **least squares**; this averages out clicking noise. (And, in the next chapter, lets **RANSAC** discard wrong matches before the final fit.)

• **a fitting analogy**: just like fitting a line $x'=ap+b$ from many noisy $(p,x')$ pairs, different correspondence sets can yield the *same* $H$; the unknowns are $H$'s coefficients, the data are the point pairs.

• **pick a reference, warp the rest**: choose one image's coordinate frame as the canvas; for every other image, warp it into that frame using its homography (warp one-by-one toward the reference). The union is the wide panorama; pixels no part of any image reaches stay **black**. [`fig-pano-warp-to-reference`]

• **inverse warp** (the right way to resample): loop over **output** pixels $(x,y)$, map *back* into the source with $H^{-1}$, then divide and sample (with interpolation/antialiasing) — this guarantees every output pixel is filled, no holes. (Forward warp leaves gaps.) The full forward-vs-inverse warp and resampling story is in [[Slopendium/09 Motion, warping, and video|Warping]].

• **bounding box**: to size the canvas, **forward**-map the source corners through $H$ to see where the warped image lands; then **inverse**-warp to fill the output. (Forward for the bbox, inverse for the pixels.)

• **then blend** — the warped images overlap and won't match exactly (exposure, vignetting, tiny misalignment); merging them seamlessly is the [[#Blending|Blending]] section's job (feathered/two-scale/Poisson). Here we just place them.

• **other projections** (pointer): warping everything onto one flat reference plane stretches badly for very wide fields of view; **cylindrical / spherical** projections (and "little planet" stereographic) are the fix — covered later under *Other projections*.

• **the planar twin**: everything above needed *pure rotation* because general scenes have depth. But if the scene is a **single plane**, a homography relates **any** two views of it — *even when the camera translates* — because a plane imposes one depth relationship across the whole surface (no independent per-pixel depth). A plane photographed from any viewpoint maps to a plane by a homography.

• **document flattening (rectification)**: photograph a page, whiteboard, painting, or building façade at an angle and it appears as a **keystoned quadrilateral**. Click (or detect) its **four corners**, set them as correspondences to a target rectangle, solve for $H$, and **warp** → a fronto-parallel, "scanned" version with verticals vertical and the page rectangular. [`fig-doc-flatten`]

• **document merging**: to capture a large document (or a whiteboard too big for one frame), shoot **overlapping** planar patches and stitch them with planar homographies — the same pipeline as a panorama, but the homography is justified by the **planar scene** rather than by pure rotation. Useful for book scanning, mosaicing microscope slides, aerial/satellite image tiles of (locally) flat ground.

• **same machinery, different justification** — worth stating for the reader: panoramas (rotation) and document rectification/merging (plane) are *the same homography solve*; only the geometric reason the homography is valid differs. (Perspective-distortion correction — making a building's verticals parallel — is the same trick and is developed in [[Slopendium/09 Motion, warping, and video|Warping]].)

• **the dead end** (worth showing): the old way matched images **densely** — try all translations, compute the sum-of-squared-differences (SSD) over all pixels, keep the best. For a **homography** that means discretising **8 coefficients** — combinatorially hopeless — and each trial requires a full warp + SSD. Even reduced to a couple of rotational DOF it's restrictive, slow, and **not robust** to moving objects or distortion. [`fig-feature-pipeline` motivates the alternative]

• **the modern idea — sparse matching**: ignore uniform regions; focus on a few **very distinctive points** (interest points), match them **locally** (which point ↔ which point), and add machinery to be **robust to local mistakes**. Sparse, fast, robust.

• **the two problems to solve** (state them as the chapter's skeleton): **(1) repeatable detection** — detect the *same* scene point **independently** in each image (if a point is found in one image but not the other, it can never match); **(2) distinctive description** — for a detected point, **recognise** its correspondent in the other image, via a descriptor that's similar for the same scene point and different for others. [`fig-corner-flat-edge`]

• **the discipline to emphasise**: detection and description run **independently in each image**. Only **at the very end**, given a bag of (point, descriptor) features per image, do we compute correspondences. We analyse the detector/descriptor on *corresponding* scene points only to *check* they'd behave consistently.

• **what makes a good keypoint**: a point you can **localise precisely** and **re-find reliably**. The test (Moravec's idea): slide a small window a little in **any** direction and measure how much the patch changes. **Flat** region → no change in any direction (not localisable); **edge** → no change *along* the edge (slides freely → ambiguous); **corner** → **large change in every direction** → uniquely pinned down. *Corners make the best keypoints.* [`fig-corner-flat-edge`]

• **quantify it**: the windowed change under a shift $(u,v)$ is $E(u,v)=\sum_{x,y} w(x,y)\,[I(x+u,y+v)-I(x,y)]^2$, with $w$ a box or Gaussian window. Computing this for all shifts is costly — so **Taylor-expand** $I(x+u,y+v)\approx I+I_xu+I_yv$ to get a **quadratic form**: $E(u,v)\approx \begin{psmallmatrix}u&v\end{psmallmatrix}M\begin{psmallmatrix}u\\ v\end{psmallmatrix}$.

• **the structure tensor** $M=\sum_{x,y} w\begin{psmallmatrix}I_x^2 & I_xI_y\\ I_xI_y & I_y^2\end{psmallmatrix}$ (a.k.a. second-moment matrix) packs the local gradient statistics into a $2\times2$ symmetric matrix. Its **eigenvalues** $\lambda_1,\lambda_2$ are the change rates along the two principal directions; the iso-$E$ contour is an **ellipse** with axes $\propto\lambda^{-1/2}$. [`fig-harris-eigen-ellipse`]

• **classify by eigenvalues**: both small → **flat**; one large, one small → **edge** (the freely-sliding direction has tiny $\lambda$); **both large** → **corner**. [`fig-harris-eigen-ellipse`]

• **the Harris response** (avoid computing eigenvalues explicitly): $R=\det M - k(\operatorname{tr}M)^2 = \lambda_1\lambda_2 - k(\lambda_1+\lambda_2)^2$, $k\approx0.04$–$0.06$. $R$ is large-positive at corners, large-negative on edges, small in flat regions. (**Shi–Tomasi** variant: just use $R=\min(\lambda_1,\lambda_2)$.)

• **the algorithm**: compute gradients $I_x,I_y$; form $M$ per pixel (windowed sums of products); compute $R$; **threshold** $R$; keep **local maxima** of $R$ (non-max suppression over a $k\times k$ neighbourhood) → the detected corners. [workflow figure within `fig-harris-eigen-ellipse` family]

• **invariances** (what survives between two views): **rotation** — the ellipse rotates but its **eigenvalues don't change**, so $R$ is rotation-invariant (a clean example of eigen-analysis giving a rotation-invariant quantity); **intensity** — uses only **derivatives**, so invariant to a brightness shift $I\to I+b$, and partially to scaling $I\to aI$ (the threshold is the catch). **Not invariant to scale**: zoom in and a corner's neighbourhood looks like an edge → it gets classified differently. That failure motivates SIFT. [`fig-harris-not-scale-invariant`]

• **the simple descriptor** (the basic, pre-SIFT version — enough for a first system): take the **$k\times k$ patch** of luminance around the corner; **blur** slightly (reduce sampling/aliasing); **subtract the mean and divide by the standard deviation** (kill brightness/contrast differences from exposure, vignetting, clouds). The descriptor is just those $k\times k$ normalised numbers.

• **brute-force matching**: for each feature in image 1, scan all features in image 2 and keep the one with smallest descriptor **SSD** $d=\lVert\mathbf{f}_i-\mathbf{f}_j\rVert^2$. This gives every corner a match — *including many wrong ones* (even points in non-overlapping regions get a "match"). Two cleanups follow: the **ratio test** (next-but-one) and **RANSAC**.

• **the goal**: a detector that finds the same point across **scale and rotation** changes, and a descriptor that's correspondingly **invariant** — so wide-baseline, zoomed, rotated views still match. SIFT (Lowe 2004) is the canonical answer. [`fig-feature-pipeline`]

• **scale selection via scale-space**: blur the image with Gaussians of increasing $\sigma$ to build a **scale-space** $L(\cdot,\sigma)=G_\sigma*I$. The right window size for a feature is found as an **extremum over scale** of a scale-invariant function. A plain average is too flat to peak; instead use a **centre-surround / contrast** operator — the **Difference-of-Gaussians** $D=L(\cdot,k\sigma)-L(\cdot,\sigma)$, which approximates the **scale-normalised Laplacian** $\sigma^2\nabla^2 G$. [`fig-scale-space-bumps`]

• **characteristic scale**: the DoG response, tracked across scale at a point, **peaks at a scale proportional to the feature's size** (the slides' 1D-bumps demo: widths 5/9/15 peak at $\sigma\approx1.7/3/5.2$ — a constant ratio). That peak scale is the feature's **characteristic scale**, found **independently in each image**, so a zoomed copy is detected at the correspondingly larger scale. [`fig-scale-space-bumps`]

• **SIFT detection**: find **local extrema of DoG in space *and* scale** — a point larger/smaller than all **26** neighbours (8 in-scale + 9+9 in adjacent scales) of a **DoG pyramid**, processed one **octave** (factor-of-2) at a time. Then **refine** to sub-pixel/sub-scale by fitting a quadratic (Brown & Lowe 2002), and **reject** low-contrast points and edge-like points (the principal-curvature / Harris ratio test). The alternative **Harris-Laplacian** (Mikolajczyk & Schmid) maximises Harris in space and Laplacian in scale. [`fig-dog-pyramid`]

• **canonical orientation** (for rotation invariance): build a **histogram of local gradient orientations** (weighted by gradient magnitude) over the keypoint's neighbourhood at its scale; the **peak** sets the keypoint's **canonical angle**. Describing the patch *relative* to this angle makes the descriptor rotation-invariant. [`fig-canonical-orientation`]

• **the SIFT descriptor** (128-D gradient-orientation histogram): resample a **$16\times16$** window in the patch's **canonical scale and orientation**; compute gradients, **weight by a Gaussian** (smooth falloff toward the edge); divide into a **$4\times4$** grid of cells; in each cell accumulate an **8-bin orientation histogram** (weighted by magnitude, with trilinear interpolation so a gradient spreads smoothly across 8 bins — 4 spatial + 2 orientation). $4\times4\times8=\mathbf{128}$ numbers. [`fig-sift-descriptor`]

• **illumination invariance**: **normalise** the 128-vector to unit length (kills contrast scaling); **clamp** any component $>0.2$ and renormalise (a few large gradients — e.g. from a specular edge — don't dominate); using **gradient orientations** is already fairly lighting-robust. Net invariances: **scale, rotation, and affine illumination** (and graceful degradation under small affine geometry).

• **why it matters** — locality (robust to occlusion/clutter, no segmentation), distinctiveness (a single feature can be matched against a database of thousands), quantity (many per image), efficiency. **Matching at scale** uses approximate nearest-neighbour structures — **k-d trees / Best-Bin-First** (Beis & Lowe) — instead of brute force.

• **forward-pointers**: SIFT features feed **structure-from-motion** at internet scale — **Photo Tourism / Photosynth** (Snavely et al. 2006), and today **COLMAP** → Gaussian splatting. The modern **learned** successors (SuperPoint, LightGlue, neural matchers) are cross-referenced in [[Deep learning]] — same detect/describe/match skeleton, learned end-to-end.

• **the problem it solves**: brute-force matching returns a nearest neighbour for **every** feature — including features with **no true correspondent** (background, non-overlap, repeated texture). We need to tell good matches from bad. [continues `fig-feature-pipeline`]

• **why an absolute threshold fails**: thresholding the raw descriptor distance ("if $d$ too big, reject") doesn't cleanly separate good from bad — the distributions **overlap**; there's no clean cutoff (good matches can be far, bad matches can be near). [`fig-ratio-test-histograms`]

• **Lowe's ratio test** (the fix): compare the **best** match distance $d_1$ to the **second-best** $d_2$. Keep the match only if $\boxed{d_1/d_2 < \tau}$ (typically $\tau\approx0.8$). Intuition: a **distinctive, correct** match is *much* closer than every alternative ($d_1\ll d_2$, ratio small); an **ambiguous** match (repeated texture, no true correspondent) has a near-tie best/second-best ($d_1\approx d_2$, ratio near 1) → reject. The *relative* test is far more discriminative than the absolute one. [`fig-ratio-test-histograms` shows the well-separated ratio distributions for correct vs incorrect matches]

• **status after the ratio test**: a set of matches that is **mostly** correct — but **not entirely**. Some outliers survive (and near-duplicate structure can still fool it). That residual error is exactly what **RANSAC** mops up next.

• **the setup**: we have point correspondences, most good (inliers) but some wrong (**outliers**), and we want the homography $H$. A plain **least-squares** fit over *all* matches is hopeless — even a few gross outliers drag the fit arbitrarily far off (squared error rewards them). We need a **robust** estimator. [`fig-ransac-pano-inliers`]

• **RANSAC** (RANdom SAmple Consensus, Fischler & Bolles 1981) — the loop:

1. **randomly pick a minimal sample** — for a homography, **4 correspondences** ($s=4$);

2. **fit the model exactly** from that sample (solve $H$);

3. **count inliers** — all matches with **small reprojection error** $\lVert\mathbf{x}_i'-H\mathbf{x}_i\rVert<\varepsilon$;

4. **keep the sample with the largest consensus set** (most inliers);

5. after the loop, **re-fit $H$ by least squares on all inliers** (optionally reweighted to pull in near-inliers). [`fig-ransac-line` shows the loop on a toy line fit; `fig-ransac-pano-inliers` on a real overlap]

• *pseudocode*: the canonical robust-fitting loop as a fence — sample 4, fit, count inliers, keep best, re-fit on inliers.

• **the intuition**: a model fit to an **all-inlier** sample is approximately correct and so will be **endorsed by many other inliers**; a model contaminated by an outlier is wrong and gets **few votes**. Consensus, not least squares, identifies the good model. The minimal sample size keeps the chance of a clean draw as high as possible.

• **probabilistic analysis — how many iterations?** Let $w$ be the **fraction of inliers** and $s$ the sample size. The probability a single random sample is **all inliers** is $w^s$ (independent draws → product / AND). The probability that $N$ samples **all fail** to be clean is $(1-w^s)^N$ (each independent → product). To **succeed with probability $p$**, demand $(1-w^s)^N \le 1-p$, giving

$$N \;=\; \frac{\log(1-p)}{\log(1-w^s)}.$$

[`fig-ransac-iterations-graph`]

• **read the three knobs** (state the takeaways):

• **sample size $s$ / model complexity** — the **bad** exponential: $w^s$ shrinks fast as $s$ grows (homography needs $s=4$, so $w^4$). More parameters → many more iterations.

• **inlier fraction $w$** — the **base** of the exponential: cleaner matches → dramatically fewer iterations. (This is why the **ratio test runs first** — it *raises $w$*.)

• **iterations $N$** — the **good** exponential: failure probability falls geometrically with $N$, so a few extra iterations buy enormous reliability.

• **worked numbers** (from the slides, to ground it): with $s=4$ — at $w=0.5$, $w^s=0.0625$, so $N=100$ → fail prob $\approx0.0016$ (1 in 600), $N=1000$ → 1 in $10^{28}$. At $w=0.3$, $N=100$ → fail $\approx0.44$, $N=1000$ → 1 in 3400. At $w=0.1$, you need $\sim10^5$ iterations to be safe. **Inlier fraction dominates.**

• **the bigger lesson**: RANSAC is a **general robust-fitting** tool (lines, fundamental matrices, planes, …) — *fit a low-parameter model from a random minimal sample, score by consensus, keep the best, re-fit on inliers.* Cheap when the model needs few points and inliers aren't too rare.

• **closing the loop — the whole auto-pano system**: extract **Harris** (or SIFT) interest points → describe (normalised patch / SIFT) → match by descriptor distance → **ratio test** rejects ambiguous matches → **RANSAC** fits the homography robustly → warp every image to the reference → **[[#Blending|blend]]**. That is automatic panorama stitching (Brown & Lowe 2007), and the same feature pipeline scales up to **bundle adjustment** ([[#Bundle adjustment]]) and full 3D reconstruction.

• the setup: by this point the images are **geometrically aligned** (homography from RANSAC, then reprojected into a common frame). Geometry is *not* the problem anymore. The problem is **color**. [`fig-blend-visible-seam`]

• the three culprits, all real and all unavoidable in a handheld pano:

• **exposure / auto-exposure drift**: the camera re-meters between shots (or you changed shutter/ISO), so the same wall is brighter in one frame than the next.

• **white-balance drift**: auto-WB shifts per frame → a color cast that differs across the seam.

• **vignetting**: every lens darkens toward the corners, so the *same* scene point is darker when it lands near a frame edge than when it lands center — guaranteeing a brightness ramp that disagrees across overlaps.

• why the eye pounces on it (the **L9** hook, full box above): a hard seam is a **discontinuity in the gradient field** — a step edge the visual system flags as a real boundary. A *smooth* brightness difference of the same magnitude, by contrast, is nearly invisible. So "hide the seam" really means "**don't leave a gradient discontinuity at the boundary**."

• encoding note (L2): do all blending in **linear light** (un-gamma first). Light from overlapping frames *adds* linearly; averaging gamma-encoded values darkens midtones and warps the blend. State this before any weighted average.

• the naive non-answer — **hard cut** ("basic reprojection"): just paint each output pixel from whichever source covers it. Result: a sharp, ugly seam exactly where exposure/WB/vignetting disagree. This is the baseline every method below beats. [`fig-blend-visible-seam`]

• the first real idea (**Solution 1: smooth blending**): make each output pixel a **weighted average** of the sources, with the weight **falling off toward each frame's boundary**: $I_{out}(p)=\dfrac{\sum_i w_i(p)\,I_i(p)}{\sum_i w_i(p)}$. A source counts fully in its interior and fades to zero at its edge, so the transition is gradual and the seam softens.

• computing the weight cheaply: the distance-to-boundary is **awkward in the output frame** (the homography warps it) but **trivial in each source frame** — a **separable, piecewise-linear** ramp (1 in the middle, 0 at the edge, in $x$ and in $y$). Compute the ramp in source space, warp it along with the image. [`fig-blend-feather-ghost`]

• engineering note worth stating (slide 77): keep an **extra "sum of weights" image** alongside the accumulator and divide at the end — it makes the normalization $\sum_i w_i$ a one-liner and handles arbitrary overlap counts.

• the catch — **ghosting and blur**: feathering averages *aligned* pixels, but registration is never perfect (parallax, moving objects, lens distortion). Averaging slightly-misaligned high-frequency detail **doubles edges** (ghosts) and **smears texture**. The wider the feather, the worse the blur. [`fig-blend-feather-ghost`]

• the diagnosis that motivates everything next: feathering uses **one transition width for all frequencies**. But the *right* width is **frequency-dependent** — low frequencies (the exposure/vignetting ramp we *want* to merge) need a **wide, smooth** blend; high frequencies (sharp detail we want to keep crisp and un-ghosted) need a **narrow, almost-abrupt** transition. One width cannot serve both. That realization is the whole ladder.

• the idea (the user's own course method, after Brown & Lowe's two-band simplification): split each source into **just two bands** and blend them **differently**.

• **low band** $L_i = G_\sigma * I_i$ (Gaussian-blurred): carries exposure, white balance, vignetting — the *slow* stuff we want to merge smoothly. **Smooth-blend** it with the feathering weights above. A wide transition here corrects exposure/vignetting differences without introducing a visible step. [`fig-blend-twoscale-split`]

• **high band** $H_i = I_i - L_i$ (the residual detail): carries sharp edges and texture. Here a smooth average is exactly what *causes* ghosting — so instead use **winner-take-all**: at each pixel keep the high-frequency detail from the source with the **highest weight**, $H_{out}(p)=H_{\arg\max_i w_i(p)}(p)$. An **abrupt** transition keeps detail sharp and avoids duplicated objects.

• recombine: $I_{out} = L_{out} + H_{out}$. [`fig-blend-twoscale-result`]

• why it works in one line: **smooth where the eye tolerates smoothness (low frequencies), abrupt where the eye demands sharpness (high frequencies).** It directly implements "match the transition width to the scale," just with two scales instead of a continuum.

• where it sits in the ladder: this is the **2-level special case** of Laplacian-pyramid blending (next). Brown & Lowe showed two bands are often *enough* for panoramas, which is why it's a great pset: most of the benefit, a fraction of the code. [`fig-blend-twoscale-result`]

• limitation (the push to multiband): two scales is a coarse quantization of "frequency." With strong exposure differences or fine repeating texture across the seam, two bands can still leave a faint transition. The principled version uses a **full pyramid**.

• the principle (Burt & Adelson 1983): generalize two-scale to a **continuum of scales** via the **Laplacian pyramid**. Decompose each source into band-pass levels $L^k_A, L^k_B$; **blend each band with a transition width matched to that band's scale**; then **collapse** the blended pyramid back to an image. Cross-ref [[Linear pyramids and wavelets]] for the pyramid itself. [`fig-blend-laplacian-bands`]

• the per-band blend (the headline equation): $L^k_{out}=m^k\,L^k_A+(1-m^k)\,L^k_B$, where $m^k$ is the blend mask at level $k$. Then reconstruct $I_{out}=\sum_k \mathrm{expand}(L^k_{out})$.

• *pseudocode*: the whole method as a fence — build two Laplacian pyramids plus a Gaussian pyramid of the mask, blend each level, collapse.

• the crucial detail — **the mask uses a Gaussian pyramid, not a Laplacian one**: $m^k = G^k(\text{mask})$. Blurring the binary mask more and more as you go coarser means the **transition gets wider at coarse scales and stays sharp at fine scales** — exactly the frequency-dependent width feathering couldn't deliver. (Slide 19: "Transition width proportional to pyramid level; decompose mask with **Gaussian** (NOT Laplacian) pyramid.") [`fig-blend-mask-pyramid`]

• why this beats two-scale: every frequency band gets *its own* transition width — fine detail switches over a few pixels (no ghosting), the coarse exposure/vignetting ramp blends over a wide region (no visible step). It's the optimal frequency-tradeoff, made automatic by the pyramid.

• **L16 callout (prefilter before downsample):** building the Gaussian/Laplacian pyramid *correctly* requires **blurring before you decimate** at each level — otherwise high frequencies **alias** into the coarse levels and the blend gets bold false low-frequency artifacts (moiré) that no amount of clever masking fixes. The pyramid's quality is only as good as its prefilter. (→ see Big lesson **L16** · prefilter before you downsample; first appears in BASIC — Resampling.) This is *why* the pyramid construction, not just the blend formula, matters.

• relation to exposure fusion (forward-ref): replace the **spatial** mask $m$ with a **per-pixel quality weight** (well-exposedness, contrast, saturation) and the *identical* machinery fuses an exposure bracket into a tone-mapped image — **Exposure Fusion** (Mertens 2007), used on phones. Same pyramid blend, different mask. (→ [[#HDR merging]], [[#Application to cell phones: HDR+ and burst imaging]].)

• the move (Pérez, Gangnet & Blake 2003): stop manipulating *pixel values* across the seam and manipulate the **gradient field** instead. Inside the region you're pasting, demand that the result's gradient match the **source's gradient**; on the boundary, demand it match the **target's values**. Then *solve for the pixels* whose gradients best satisfy that. This is the **same solve as seamless cloning** in [[Poisson image editing]] — here the "source region" is one panorama image and the "background" is the other. Cross-ref the worked single-image version. [`fig-blend-poisson-vs-pyramid`]

• the objective (the headline equation): $\displaystyle \min_f \iint_\Omega \lVert\nabla f - \mathbf v\rVert^2$ subject to $f|_{\partial\Omega}=I_{\text{target}}|_{\partial\Omega}$, where $\mathbf v$ is the **guidance field** (the pasted source gradient). Its Euler–Lagrange condition is the **Poisson equation** $\nabla^2 f = \operatorname{div}\mathbf v$ — a large **sparse linear system**, solved matrix-free with conjugate gradient (never forming the matrix; just repeated Laplacian convolutions — [[Efficient solvers]], and the deblurring solver of [[Single-image computational photography#Blind deblurring]]).

• why it kills the seam (the **L9** payoff): the visual system reads **gradients**, and we've made the gradients across the boundary **continuous by construction**. Any **constant brightness/color offset** between the two frames is simply **spread out and absorbed** as an invisible low-frequency shift (in 1-D: a linear ramp interpolating the boundary mismatch). The seam *vanishes* because there's no gradient discontinuity left to see — even when exposures differ substantially.

• **Poisson vs. pyramid — the DC-handling contrast** (cross-ref the single-image treatment in [[Poisson image editing]] → *Poisson vs. pyramid blending*): both hide the seam, but they handle the **DC (overall level) difference** differently. **Pyramid blending** blends the coarsest (DC-ish) band over a wide transition — it *averages* the level mismatch, which can leave a faint low-frequency halo and a compromise brightness. **Poisson** doesn't average the level at all; it **pins the boundary to the target and lets the interior float**, re-deriving absolute values from gradients — so it can shift the pasted region's whole brightness to match, but it also *inherits the target's lighting* (good for compositing, sometimes wrong for panoramas where you want to keep a source's own exposure). [`fig-blend-poisson-vs-pyramid`]

• practical caveats to surface (slides 150–152): Poisson assumes the two backgrounds are **photometrically similar**; pasting from dark into bright loses **contrast**, because Poisson reproduces *linear* differences while contrast is *multiplicative* — the fix is to do the Poisson solve in a **log / perceptual color space** (or use covariant derivatives, à la the Photoshop healing brush). State this as the honest limit. (L1/L2: contrast is multiplicative → work in log.)

• gradient-mixing bonus: you can take, per pixel, the **stronger** of source vs target gradient (max) instead of always the source — useful for blending over textured backgrounds (slide 137).

• the reframing (Agarwala 2004; Kwatra 2003): all the methods above blend over a **fixed** seam (wherever the overlap is). Seam optimization asks the prior question — **where should the seam even be?** Choose the boundary to thread through pixels where the **two images already agree**, so there's barely anything to blend. [`fig-blend-seam-graphcut`]

• the formulation as **discrete optimization on a graph**: label each output pixel with the **source it comes from**, $\ell_p$. Minimize $E=\sum_p D_p(\ell_p)+\sum_{(p,q)}V_{pq}(\ell_p,\ell_q)$ — a **data term** $D_p$ (must lie inside the chosen source / respect coverage) plus a **smoothness term** $V_{pq}$ that **penalizes a label change between neighbours by how much the two sources disagree there**. Minimizing $E$ is a **min-cut / max-flow** (graph cut) — globally optimal for two labels. Cross-ref [[Seam optimization]] for the machinery.

• **L12 callout (edits as discrete optimization on a graph):** the optimizer (graph cut) is off-the-shelf; **the energy is the entire design.** Make $V_{pq}$ = "color disagreement between the sources at this edge" and the min-cut automatically prefers to cut **where the images match** — e.g. along a uniform sky, *not* through a person's face. This is the **affinity principle (L4)** turned into a *cut* ("don't cut here") rather than an average. (→ see Big lesson **L12** · image edits as discrete optimization on a graph; first appears in [[Seam optimization]].)

• how it composes with the rest of the ladder: seam optimization decides **where** to put the boundary; you then still run a **Poisson or pyramid blend across that optimal seam** to mop up any residual exposure mismatch. Graph-cut + Poisson is the canonical photomontage compose (Agarwala 2004): **cut, then reconstruct.**

• the natural bridge to the next chapter: because the seam can route *around* a region, graph cut is also how you handle **moving objects** — pick the seam (and thus one source) so a walking person is taken *entirely* from one frame, never half-and-half. That's **deghosting**, picked up in *Bells and whistles*.