At first, the digital revolution merely replaced film by digital sensors more amenable to quick snapshots, instant gratification and sharing. But it did more than that : it created the opportunity to add arbitrary computation between the photons in a scene and the final image.

This allows us to do really cool stuff : manage scenes where contrast is excessive, computationally remove blur, combine images into virtual wide angle panoramas, and even capture the full array of light to do stuff like refocusing after the fact and extract 3D.

[results-montage figure — HDR tone mapping (Durand et al.), refocus-after-capture (Ng et al.), motion deblur (Levin et al.), panorama (problem set)]

In fact, even a simple snapshot requires serious computation because each pixel usually captures only one of the 3 channels and the other two need to be reconstructed [demosaick-snapshot figure]. Cell phones these days take much better images than they have a right to because of computational photography.

Plus generative AI is poised to further revolutionize photography and image making with its ability to learn the space of all possible images in a manner reminiscent of a Borges novella (cite *The Library of Babel* and https://arxiv.org/abs/2310.01425)

• the book is organized **cross-cuttingly**: a chapter is sometimes a **tool** (convolution, the bilateral filter), sometimes a **big idea** (priors, the light field), and the major themes deliberately **recur** across chapters — the same lesson seen from several angles. The redundancy is intentional; cross-references thread the pieces together.

• **hammers and nails**: we deliberately sought a **balance between "hammers and nails"** — between **tools/techniques** (the hammers: convolution, the bilateral filter, optimization, priors) and **applications/challenges** (the nails: HDR, panoramas, deblurring, refocusing). Rather than a purely tool-first or purely application-first table of contents, the book's structure is **organized by a combination of tools and applications/challenges**, alternating between the two so each motivates the other.

• **intuition first**: lead with the idea and a picture, then the math. The audience is a **CS undergrad**, new to imaging (what to know coming in is spelled out in *[[#Prerequisites]]* below).

• **how to read**: FUNDAMENTALS grounds the physics and perception; BASIC IMAGE PROCESSING gives the toolkit; later parts build single- and multi-image computational photography. Read mostly front-to-back, or follow the dependency graph below to chart your own path.

• **how this book came to be**:

• grew out of Frédo's **computational photography course** at MIT — its lectures, slides, and problem sets are this book's backbone.

• written **with an AI collaborator** (Claude) as a deliberate experiment in **AI-assisted authoring** — the outline, figures, drafts, and the book's own tooling were co-developed; a fitting, slightly meta nod to the generative-AI theme the book itself discusses.

• the **process** (how outline, figures, and prose are generated, verified, and kept consistent) is documented in an appendix — itself a small piece of AI methodology [forward ref: "how this book was created"].

• **chapter dependency graph**:

• *(placeholder — to create later)* a **graph of chapter dependencies** belongs here, at the end of the intro: a diagram showing which chapters rely on which, so a reader can chart their own path through the book. Generate it from the **provides / requires** graph (see [[Book Generation Process#Cross-chapter dependencies]] and each section's `- prereqs:`), once enough of the book is written for the dependencies to be stable.

• a **short, intro-level** chapter (the headline version of the appendix [[Appendices#How this book was created (the early process)]]): the book is partly *about* generative AI and was partly written *with* it, so the making-of is part of the story, not a footnote.

• **where it came from**: grew out of Frédo's MIT computational photography course — ~**305k words** of auto-captured/auto-transcribed lecture transcripts, ~**114k** of slide decks, ~**42k** of problem sets (~**460k** of pre-existing teaching material) form the backbone.

• **how it was written**: drafted **with an AI collaborator** (Claude); outline/prose split with a compile step; figures are programs; the human edits triage back so re-compiling reproduces the corrections.

• **who did what**: AI for laborious-but-specified drafting, figure code, and bookkeeping; human for structure, judgment, catching subtle errors, and ownership of every claim.

• **the numbers, honestly**: ~**40.5k words** of typed prompts steered ~**735k words** of generated outline+prose (≈13–18× leverage, *not* authorship); the outline is model output too, so the earlier "2.2× human writing" framing was wrong — the real human contribution is **curation**, not keystrokes. Point to the two appendices for the full account and the under-the-hood prompts/verifiers.

• the book is **language-agnostic in its ideas** but ships in two editions — a **Python** one and a **C++** one (never one book showing both) — because the two languages serve different moments: Python to *think and prototype*, C++ to *ship and run fast*. The concepts are identical; pick the edition that matches your goal.

• **why choose which**: **Python** for **deep learning, prototyping, and convenience**; **C++** for **performance**. The gap is not small. When you write the **low-level code yourself** (e.g. your own convolution rather than a library call), the **speed differential** is dramatic: in Frédo's experience, roughly **1000× from Python to optimized C++**, and **at least another ~10× from C++ to Halide**. So the language choice is also a performance choice — see [[Making computational photography code fast]] and [[Appendices#Programming: Python, C++, and PyTorch]].

• **Python** — the right default for *learning, prototyping, and research*:

• **pros**: **readable** — NumPy array code reads almost like the math (`out = a + 0.5*b`); **fast to write and explore**, with interactive notebooks and instant feedback; a vast **ecosystem** (NumPy, SciPy, Pillow / OpenCV, scikit-image, Matplotlib, and the whole deep-learning stack — PyTorch, JAX); automatic memory; the lingua franca of research and ML.

• **cons**: **slow at the pixel level** — a naive per-pixel Python loop is orders of magnitude slower than C, so you must **vectorize** (push the loop down into NumPy's C core) or it crawls; the **GIL** limits real CPU threading; packaging / deployment is awkward; dynamic typing lets type bugs hide until run time; little control over memory layout or exact performance.

• bottom line: the right default for **learning, prototyping, and research** — and, through NumPy / PyTorch, fast *enough* for most of what we do here.

• **C++** — the right default for *performance and production*:

• **pros**: **fast and predictable** — close to the metal, with control over memory layout, cache behavior, and SIMD; what real **camera ISPs, real-time pipelines, and shipped products** are written in; static types catch errors at compile time; deploys to phones and embedded hardware.

• **cons**: **verbose and slower to write**; **manual memory management**, a rich source of bugs (out-of-bounds, leaks, use-after-free); a heavier **build / toolchain** story; far less interactive (edit–compile–run, not edit–run). The productivity tax is real.

• bottom line: the right default for **performance and production** — once the algorithm is settled and it must run fast, on device, at scale.

• **Halide** — a **domain-specific language** (embedded in C++ / Python) for high-performance image processing: you write the **algorithm** (what each pixel computes) **separately** from the **schedule** (how to tile, vectorize, parallelize, and fuse it across the memory hierarchy).

• **pros**: near-hand-tuned speed **without** hand-writing the tangled, hardware-specific loops; the *same* algorithm retargets to CPU / GPU / DSP by swapping the schedule; the basis of real camera stacks.

• **cons**: another language and mental model to learn, best reserved for the **hot paths** you have already identified — premature scheduling is its own trap. Covered properly in [[Making computational photography code fast]].

• **MATLAB (historical note)**: for years *the* platform for this work — a strong numerical/linear-algebra library and an IDE that handled **images natively** (an image is a matrix; `imread`/`imshow`/slicing; the Image Processing Toolbox); much of the literature was prototyped in it. **Largely displaced by Python** (NumPy fills the same array niche, free) and especially **PyTorch** (autodiff + GPU + deep learning, which MATLAB never matched). Worth knowing because older papers/code are in it, and its array-thinking carries over to NumPy.

• **Libraries (implement-from-scratch, then build on)**: the book has you **implement core algorithms from scratch** to understand them (the `imageops` reference), but to *build* / dive into advanced projects, stand on mature libraries — **Python**: Pillow (PIL), NumPy/SciPy, OpenCV, scikit-image, PyTorch (also a GPU-array + autodiff engine); **C++**: OpenCV (de-facto standard), Eigen (linear algebra), LibRaw (raw decode), Halide (hot paths).

• **side bar — Photoshop 1.0, in 128,000 lines (a language lesson from 1990)**: Adobe (via the **Computer History Museum**, 2013) released the source of **Photoshop 1.0.1** — **≈128,000 lines across 179 files**, written for the Macintosh by **Thomas Knoll** (its sole engineer for v1 — a computer-vision PhD student whose 1987 image-display hack became the app). The language split *is* this section's lesson, 35 years early: **≈75 % Pascal** (the productive high-level language) and **≈15 % hand-written 68000 assembly** for the speed-critical inner loops — *most of it readable, the hot pixels hand-tuned* — exactly the Python-vs-C++/Halide trade-off (a high-level language for the bulk, machine code where it counts; what **Halide** now automates). The code is famously *mostly uncommented but well-structured*. It's now studied as a cultural artifact: the **"Das digitale Bild"** project (DFG / LMU Munich) runs an ongoing **Critical-Code-Studies** *"reading the source code of Photoshop"* — code as a text to be close-read. **For scale:** today's Photoshop is reportedly **millions** of lines of C++ (Frédo's estimate ~**10 M**); **Adobe Camera Raw (ACR)** and **Lightroom** share the same Camera Raw engine, and a typical camera **ISP** is a large proprietary C/C++ codebase — exact modern sizes mostly **not public** [⚠️ confirm the ~10 M / current ACR/Lightroom/ISP figures]. (→ the hand-tuned-inner-loop angle returns in [[Making computational photography code fast]].)

• **Vibe coding (LLM-assisted)**:

• writing image code with an **LLM** ("vibe coding") is now part of the toolkit: great for **boilerplate, I/O, visualization, and explaining errors**, and a quick way to draft a function or a test.

• **but** image code is **numerically and perceptually subtle** — "looks plausible" is not "correct" — so keep the model on a **short leash** for filtering, resampling, color, and antialiasing, and always **verify against a hand-checkable input** (a constant, an impulse, a half-plane) before trusting it. Treat generated image code as *guilty until tested* (see [[Developing, Testing and Debugging]]). AI coding sometimes **fumbles the trivial** even when it nails the hard parts — see the cross-fade anecdote in [[Basic image processing and ISP#Vibe coding: writing image code with an LLM]].

• **PS0 — Environment & C++ basics**: toolchain setup, a minimal `Image` class, image I/O. (→ [[Developing, Testing and Debugging]])

• **PS1 — Image class, point operations, color**: pixel access, brightness/contrast, luminance–chrominance (YUV), gamma, white balance, the Spanish-Castle illusion. (→ [[Image representation]], [[Tone point operations]], [[Perceptual color and trichromatic vision]])

• **PS2 — Convolution & the bilateral filter**: box/Gaussian blur, gradients, sharpening, and bilateral denoising (with a YUV variant). (→ [[Convolution]], [[Bilateral filtering]])

• **PS3 — Denoising & demosaicking**: align-and-average a burst, ISO/variance; Bayer demosaicking (green-first, edge-based, color-difference) and Prokudin-Gorsky channel alignment. (→ [[Slopendium/Denoising]], [[Slopendium/Demosaicking and joint methods]])

• **PS4 — HDR**: merge an exposure bracket into a radiance map and tone-map it. (→ [[HDR merging]], [[Global tone mapping]])

• **PS5 — Resampling, warping & morphing**: nearest/bilinear/bicubic/Lanczos resampling; Beier–Neely segment warps; a full Beier–Neely morph. (→ [[Warping]], [[Morphing]])

• **PS6 — Homographies & manual panoramas**: homogeneous coordinates, warp by a homography, solve $H$ from four correspondences, stitch a planar mosaic. (→ [[Manual panorama stitching from multiple views]])

• **PS7 — Automatic panoramas**: Harris corners (structure tensor), patch descriptors, the second-nearest-neighbor ratio test, RANSAC, and feathered / two-scale blending. (→ [[Automatic panorama stitching from multiple views and feature matching]])

• **PS8 — Non-photorealistic rendering**: paintbrush splatting; single-scale, two-scale, and gradient-oriented painterly rendering. (→ [[Non-photorealistic rendering]])

• **PS9 — Make-your-own / video & ethics**: a self-proposed project (optical-flow retiming, phase-based video magnification, …) plus the ethics deliverable; its open menu of advanced topics seeds many [[Computational optics and coded imaging]] sections. (→ [[Optical flow]], [[Motion and video magnification|Video magnification]])

• chapter intro: a working model of light — its **color** (spectrum), its **interaction with surfaces**, and its **energy** (radiometry); closes with the **plenoptic function** that ties the pipeline together

• sensor alone would capture a diffuse light. We need to select which rays from which part of the scene contribute to which pixel.

• chapter intro: from the **pinhole** (sharp but dim) to a **lens** (bright but with one focus plane) — refraction at curved surfaces, the thin-lens linearization, and the depth-of-field that the finite aperture forces on us

• chapter intro: the unifying idea behind sensing — **measurement = integrating a slice of the plenoptic function** — then the physical sensor that does it (photosites, CCD/CMOS) and the **noise** that limits it

• sidebar — **photography as objective measurement**: the medium's long arc bends toward objective measurement (film/print = a *performance*; → measurement). The **digital sensor** is the far end: because each photosite *counts photons*, a **raw** image is ≈ a **calibrated radiometric measurement** — a physical map of scene radiance, in real units, of one snapshot of time — not just a rendering. This is what makes HDR / depth / deblur possible: the pixel numbers must *mean* something physical. Ties to the intro's **generation → measurement → generation** arc (→ [[Intro]]).

• **defocus / circle of confusion**: a point off the focus plane images to a point that is not on the sensor, so the aperture cone meets the sensor as a blur disk — the **circle of confusion**. A point still counts as sharp while its disk stays within an acceptable c. [CoC figure]

• the blur disk grows with the **aperture** (wider → bigger disk) and with the **object's distance from the focus plane** (zero exactly at focus); focal length and focus distance enter too. [defocus-vs-parameters figure]

• **depth of field** = the *range* of object distances whose blur disk stays within c; the **near** and **far** limits are where the blur just reaches c. [DoF near/far figure — each cone carried out to the CoC at the sensor]

• **deriving the limits**: on the image side a near-limit object images *behind* the sensor and a far-limit object *in front* of it; **similar triangles** relate the aperture D, that convergence distance, and the circle of confusion c — and together with the conjugate relations (1/f = 1/s + 1/v) they fix the near (front) and far (back) object-distance limits. [near-limit and far-limit similar-triangle figures]

• **hyperfocal distance** H = f²/(N·c): focus there and everything from **H/2 to infinity** is acceptably sharp — the focus setting that maximizes DoF (focusing at ∞ instead wastes the H/2…H zone). [hyperfocal figure]

• DoF **grows** with a smaller aperture (larger N), a shorter focal length, and a greater focus distance — these are the paraxial (linearized) formulas; the accurate model is in the Optics part. [DoF-vs-parameters figure]

• **the surprising invariance** (the lecture's flagged "important conclusion"): for a **fixed framing** (same subject size) and a **fixed f-number**, the depth of field in **object space** is essentially **independent of focal length** — switching 28 mm → 100 mm forces you to step back, and the larger physical aperture cancels the longer focal length. So "long lenses have shallow DoF" is really about **magnification**, not focal length per se. (The *background* still looks more blurred with the long lens — that's magnification of the out-of-focus disk, not a DoF change.) [same-framing/same-N → same DoF figure]

• ⚠️ **but the invariance is only *first-order* — the background gives the focal length away.** "Same framing + same f-number → same DoF" is the **small-defocus (paraxial)** result: it holds for the **near/far acceptable-sharpness limits straddling the subject**, but **not** for how hard a *distant* background is thrown out of focus. **Derivation** (thin lens; blur-disk diameter on the sensor): a point at distance $s_b$, with the lens focused at $s_f$, images to a circle of confusion $c \approx \dfrac{f^2}{N}\,\dfrac{|s_b-s_f|}{s_b\,s_f}$. **Fix the framing** — subject magnification $m=f/s_f$ held constant, so $s_f=f/m$ — and **fix $N$**, with the background a gap $\Delta=s_b-s_f$ behind the subject:

$$c(f)=\frac{m^2\Delta/N}{\,1+ m\Delta/f\,}\ \xrightarrow[f\to\infty]{}\ \frac{m^2\Delta}{N}.$$

The subject-zone DoF is ~independent of $f$ (the leading $f$ cancels — the **first-order invariance**), but the **background blur grows monotonically with focal length**, saturating at $m^2\Delta/N$. Two shots matched in framing *and* aperture render the **subject** identically, yet a **200 mm melts a far background far more** than a 35 mm: the correction term is order $m\Delta/f$ — negligible for a background near the focus plane (invariance looks exact), but dominant once the background is many focus-distances away. This is exactly why portrait shooters reach for long lenses for "background separation" even though the DoF on the *face* is unchanged. **Full derivation + plots** of $c$ vs. background distance for 35 / 85 / 200 mm (fixed framing & $N$) — the curves coincide near the focus plane and fan apart far from it. [`fig-dof-background-blur`]

• **object-space picture & a caveat**: the simplest view is a **double cone** in front of the lens whose waist sits on the focus plane — it gives the blur for **one** scene point. But **defocus is *not* a spatially-invariant convolution**: the circle of confusion depends on each point's **scene depth** (not its pixel location), and at depth discontinuities **occlusion** matters — which is why naive "just blur the image" fakes look wrong, and why light-field / depth-aware methods (Advanced) do better. [object-space double cone; depth-dependent-blur + occlusion figure]

• **sensor size scales DoF**: shrink the sensor (keeping FOV) and both the circle of confusion and the aperture shrink, so DoF grows **linearly** with sensor size — why phone cameras have huge DoF (and resort to *computational* background blur), and why **macro is easier on small sensors** (a scaled-down camera photographs a scaled-down world — modulo diffraction)

• 🖱️ **Interactive (web edition):** a live **macro depth-of-field simulator** — a to-scale thin-lens diagram, a lens-supersampled 3D photo (real bokeh), and a circle-of-confusion-vs-distance plot all driven by one shared optics model (focal length, f-number, sensor, focus distance, subject–background gap). A 3D bug (beetle / bee / ladybug) over a flower makes the macro lesson vivid: focus lands on the front of the face and the rest of the body, antennae, and background all fall out of focus. [fig-depth-of-field-sim; interactive] *(bug & flower 3D models generated with Meshy AI — acknowledge in the caption)*

• **blur from subject motion**: $b_{px}\approx f_{px}\,v\Delta t/D$ — a straight streak = image velocity × exposure (constant-velocity linearization); only cross-axis motion blurs

• **blur from camera shake**: translation $b\approx f_{px}\Delta_c/D$ (depth-dependent); rotation yaw/pitch $b\approx f_{px}\theta$ (**depth-independent**, small-angle); roll = spin about center, grows with radius

• **which dominates**: ratio $\theta D/\Delta_c$ → rotation wins at normal distance, translation only up close (macro); rotation blur is **spatially non-uniform** so one kernel can't undo it (Whyte); IMUs measure the rotation to deblur (Joshi)

• **hand-holding rule** $\Delta t\lesssim 1/f$ s (×crop; ~2× faster for dense sensors); **image stabilization** cancels camera *rotation* (buys 3–5 stops) but not *subject* motion

• 🖱️ **Interactive:** motion-blur calculator — 3-D scene (Meshy walking human / car), temporal-supersampled blur, world+screen readout by source, motion-vector & flow-color overlays, hover pixel scale [fig-motion-blur-calculator; interactive]

• the noise **sources** (their variances add):

• **photon / shot noise** — the Poisson statistics of *counting photons*: variance = mean, so σ = √N. Fundamental (it's the light itself), dominates the midtones/highlights; averaging N independent frames cuts it by √N.

• **read noise** — added by the amplifier / ADC on readout; signal-independent, dominates the **shadows** and sets the noise floor (hence dynamic range).

• **thermal / dark current** — electrons freed by heat, independent of light; grows with exposure time and temperature (long exposures & astrophotography → sensor cooling, dark-frame subtraction).

• **fixed-pattern / pattern noise** — per-pixel gain/offset non-uniformity (PRNU/DSNU), hot/stuck pixels, banding; *structured*, so it reads as worse than random noise of equal magnitude, and is removed by calibration (flat / dark fields).

• 💡 **Big lesson — in a *linear* image, noise variance is an *affine* function of brightness.** The variances add: shot noise gives a term **proportional to the signal** (Poisson, variance = mean, scaled by the gain) and read noise a **constant** floor, so the total is **variance ≈ gain·signal + read²** — a straight line in brightness. You can *measure* it: take an aligned **burst** of a static scene and, per pixel, plot the variance across frames against the mean — the points trace that line, slope = photon gain, intercept = the read-noise floor. And a single pixel's value over many frames is **≈ Gaussian**, which is what licenses additive-Gaussian noise models. [`fig-noise-affine` — per-pixel variance vs brightness, measured from a 50-frame ISO-3200 burst; `fig-noise-gaussian-pixels` — per-pixel histograms] → register [[Big Lessons]]

• 💡 **Big lesson — noise is *clipped* at black and white, so near the extremes it is *not* zero-mean.** A recorded value is clamped to [0, max]: near black the negative half of the noise is cut off (frames pile up at 0), near white the positive half is. So at the extremes the noise distribution is **asymmetric** and its mean is pushed **inward**. The consequence for **denoising**: any naive average/smoothing returns that biased mean, so it makes **shadows come out too bright and highlights too dark** — a bias every denoiser has to correct for (carried forward to [[#Denoising basics]] in BASIC). [`fig-noise-truncation` — a near-black pixel pinned at 0 a third of the time, its average biased bright; from the ISO-3200 burst] → register [[Big Lessons]]

• **the key intuition — it's about ratios (SNR)**: shot noise is **Poisson**, so σ = √N grows with brightness — measured noise is actually **higher in the highlights**. Yet noise *looks* worst in the **shadows**, because perception cares about the **signal-to-noise ratio** SNR = N/√N = √N, which is worst where N is small. "Ratios are all that matters" — the same reason we encode in gamma/log, and the motivation for ETTR (below — Photography 101). [std-dev map vs SNR map figure]

• **dynamic range** = the brightest recordable signal (sensor **saturation / full-well**) ÷ the **noise floor** (read noise in the shadows); it sets how much of a high-contrast scene fits in a *single* exposure → the motivation for **HDR** (Multiple exposure)

• **HDR (high dynamic range)**, sometimes called **wide dynamic range (WDR)**, is a deliberately **fuzzy** term — flag this. It gets used for at least four different things: the *scene* (a high-contrast world), the *capture* (multi-exposure / a high-DR sensor), the *file/encoding* (float or ≥10-bit radiance, e.g. OpenEXR, HDR10), and the *display* (an HDR monitor) — and, loosely, for the **tone-mapped "HDR look."** There is no single sharp definition; say which sense you mean. (Developed in [[Multiple exposure imaging]].)

• 💡 **Big lesson (L2.48) — dynamic range is set by full-well capacity over the noise floor**: a single exposure records from a **top** (the photosite's **full-well capacity** — where it clips to white) down to a **bottom** (the **noise floor** — read noise in the shadows); their **ratio is the dynamic range** (quoted in stops). You **widen** it with a **bigger well** (larger photosites — why a full-frame sensor out-ranges a phone) or a **lower floor** (cooling, lower read noise), and you **beat** it altogether by merging **multiple exposures** (HDR). The capture-side companion to **L2.49**: what bounds a shot is this *range*, not the bit depth. → register [[Big Lessons]]

• **examples, in stops** (the figure): a color **slide** is narrow (~5–6 stops), a reflective **print** ~6–7, **film negative** wide (~12–13 latitude), a **phone** sensor ~10–12, a **full-frame** sensor ~14; the **human eye** is ~10–14 stops *instantaneously* but ~**20+** once **adaptation** is allowed — and some **animals** do better in their niche; an outdoor sun-and-shadow scene can exceed **20 stops**, which is why no single capture holds it [fig-dynamic-range-comparison]

• **see it — recover an underexposed shot, full-frame vs phone**: shoot the *same* underexposed scene as **raw** on a **full-frame** camera and on a **phone**, then **push the exposure up** in software (e.g. +3–4 stops). The full-frame frame cleans up (deep full-well + low read-noise floor → wide dynamic range), while the phone frame breaks down into **shadow noise and banding** as the lift amplifies its read-noise floor — a direct, visible demonstration that **dynamic range scales with photosite / sensor size**, and exactly why phones lean on **burst / HDR+** (Multiple exposure) rather than a single exposure. [fig-underexposure-recovery-ff-vs-phone; photo — paired Fredo full-frame DNG + phone raw] *(also a coding exercise — see [[Exercises & Experiments]])*

• 💡 **Big lesson (L2.49) — quantization is rarely the real problem**: with enough bits and a sane encoding, it's **noise and dynamic range** (this section) that bite — not the number of levels. The one exception is **gamma / banding in shadows**, which is exactly why gamma encoding exists (allocate codes perceptually); so "rarely" ≠ "never". Reminded again in BASIC → Image representation → *Float vs 8-bit*.

• light, lens, pupil, retina,

• cones/rods, distribution, fovea — note there are **no S (blue) cones in the very center of the fovea**, and far more L+M than S overall, so **blue is low-resolution and a bit out of focus** (chromatic aberration) — yet we perceive blue everywhere (the brain fills it in). Luminance ≈ the **L+M cone sum**, which is why **green carries most of the acuity** (and why Bayer uses 2× green).

• cells, ganglions, on-retina spatial processing, visual cortex, V1, higher.

• **the eye as a camera — and its refractive errors.** Like any lens system the eye can mis-focus: **myopia** (eyeball too long / cornea too curved → image forms *in front* of the retina), **hyperopia** (too short → *behind*), **astigmatism** (a non-spherical cornea → two focal lines, not one point), and **presbyopia** (the aging lens stiffens and loses **accommodation**). Each is corrected by a lens quoted in **diopters** (D = 1/focal-length in meters): a negative/diverging lens for myopia, positive/converging for hyperopia, a **cylindrical** lens (with an axis) for astigmatism. [`fig-refractive-errors`]

• **how the prescription is *measured* (the machines), and why it's just inverse imaging.** **(1) Retinoscopy** — shine a moving streak of light into the eye and watch the **retinal reflex** sweep across the pupil; its direction and speed reveal myopia vs hyperopia, and trial lenses are stacked until the reflex stops moving ("neutralization"). **(2) Autorefractor** — automates this objectively: it projects an **infrared** target onto the retina and analyses the **returning image** (its focus/size, or a Hartmann–Shack / Scheimpflug pattern) to solve for the corrective lens that makes the retinal image sharp — essentially a little camera running autofocus *on your eye*, in seconds. **(3) Phoropter** — the familiar "better one… or two?" wheel of trial lenses over an eye chart: a **subjective** refinement of the machine's objective estimate. **(4) Wavefront aberrometer (Shack–Hartmann)** — for the full optical picture: a **lenslet array** samples the wavefront leaving the eye and measures **higher-order aberrations** (coma, spherical, trefoil), not just sphere/cylinder — the same wavefront sensing used in **adaptive optics** and in planning **LASIK**. The throughline: measuring an eye is **inverse imaging** — find the optics that would make the retinal image (or the returning wavefront) ideal. (Cross-ref wavefront / Shack–Hartmann sensing and adaptive optics in [[Slopendium/Compound lenses, and aberration correction]].)

• the perceptual machinery the visual system runs *on top of* color: adapting to level, working in ratios (contrast), and discounting the illuminant to report surface reflectance (lightness and color constancy), the properties compression, tone mapping, and display design optimize for.

• **the eye evolved as a sequence of small improvements**, and it recapitulates the optics of this book — from "no image" to a focused one. Following Nilsson's stages (Land & Nilsson, *Animal Eyes*):

• **a flat patch of photoreceptors** (an *eyespot*): just **photosites**, **no image formation** — it tells light from dark and, crudely, which side has *more* light (phototaxis), but cannot form a picture. This is the bare **sensor** with no optics.

• **a concavity / cup**: fold the patch into a **pit** and the rim's shading gives **directional** sensitivity — the deeper the cup, the better you can tell *where* light comes from.

• **a pinhole**: close the cup almost shut → a **pinhole eye**, a real (if dim) **image** with no lens at all (the **nautilus** still uses one) — exactly the camera obscura of [[#Pinhole image formation and linear perspective]].

• **a lens**: fill the aperture with a **refractive lens** (often a graded-index sphere) → a **bright, focused** image, independently evolved in fish, cephalopods, and vertebrates. Add a **cornea, iris, and accommodation** and you have the human eye.

• so the human eye is *one* solution; evolution reached the same goal by very different routes (below).

• **a tour of those other solutions** — a useful contrast to the human eye and to camera design:

• **compound eyes** (insects): many **ommatidia** → wide field of view and fast temporal response, but low spatial resolution

• **more cone types**: birds / reptiles are **tetrachromatic** (a 4th cone, into the UV); the **mantis shrimp** has ~12–16 photoreceptor classes (yet poor color *discrimination* — it seems to recognize rather than compare) — the **divergence of opsins** across animal groups is the color sidebar in [[#Color]]

• **tapetum lucidum** (cats, deer): a reflective layer behind the retina that boosts night vision (and causes eye-shine)

• **polarization vision** (cephalopods, many insects): they see light's **polarization** — a channel we're blind to

• **other eye designs**: **pinhole** (nautilus), **mirror eyes** (scallops — image formed by a concave mirror, not a lens), and the front-facing (**stereo**) vs side-facing (**field-of-view**) placement trade-off

• further reading: Land & Nilsson, *Animal Eyes*; the "Evolution of Eyes" chapter in *Vision* (Cambridge); a friendly overview at phos.co.uk, "The Evolution of Sight".

• four ways to turn a monochrome sensor into a color one: time, space (Bayer), prism (3-CCD), depth (Foveon), plus spectral/Lippmann and hybrids

• trichromatic vs spectral (multispectral) capture

• one 3-D space (human vision → 3 numbers); every color space is coordinates on it

• short sections: cones/LMS/metamerism; XYZ; chromaticity+gamut; linear RGB & why many; white point & balance; gamma for encoding; sRGB; RAW; log for HDR/video; opponents; HSV for UI; perceptually uniform for differences

• then two reference sections folded in: **Color and encoding in the real world** (ISPs, PyTorch/OpenCV, comp-photo pipelines, computer graphics, gamma & interpolation) and **What color space and encoding to do what?** (the task→space table and the by-space list)

<!--uid: cameras-->

<!--tags: background useful hardware standard-->

<!--uid: exposure-->

<!--tags: background useful practice-->

<!--uid: lenses-->

<!--tags: background useful practice standard-->

<!--uid: illumination-and-the-flash-->

<!--tags: background essential practice standard-->

<!--uid: traditional-and-digital-darkroom-->

<!--tags: background useful practice standard-->

<!--uid: displays-->

<!--tags: background useful hardware standard-->

<!--uid: types-of-photography-->

<!--tags: background useful practice standard-->

<!--uid: limitations-of-the-medium-->

<!--tags: background optional practice perception standard-->

<!--uid: photography-practitioner-tldr-->

<!--tags: background useful practice standard-->

• promoted from a section of [[#Fourier]] into its own chapter (it earns one): a digital image is a **sampling** of a continuous scene, and detail finer than the grid **aliases** — folds down to a wrong, low frequency (moiré, wagon-wheel).

• **aliasing = a high frequency in disguise** (sine sampled below 2×/cycle → looks like a slower sine); 2-D version = moiré on a facade. Figures: `fig-aliasing`, `fig-aliasing-photo`.

• **Nyquist–Shannon**: sample at >2× the highest frequency; the **Nyquist frequency** is half the sampling rate; above it folds down irreversibly.

• **frequency-domain mechanism**: sampling is **multiplication by a Dirac comb** in space, so by the convolution theorem it becomes a **convolution in Fourier** — but by an unusually *simple* spectrum. The comb is **self-dual**: the Fourier transform of a Dirac comb is **another train of Diracs** (spaced at the sampling rate). And that is exactly why sampling is *easy* to reason about in Fourier: convolving by a single Dirac is just a **shift**, so convolving by a *train* of Diracs is just a **sum of shifted copies**. The spectrum is simply **replicated** at every multiple of the sampling rate — no smearing, just shifts. (This is the payoff of Fourier being built on the **shift theorem**: the one convolution that would otherwise be messy collapses into pure translation.) Band-limited copies sit apart; too-wide copies **overlap** = aliasing (`fig-sampling-comb`). Big lesson **L4.13**.

• **cure**: low-pass **pre-filter** (blur) *before* downsampling. Interactive demos: `fig-sampling-1d-demo` (1-D sampling/aliasing/pre-filter, tick-marked samples + spectrum) and `fig-sampling-2d-demo` (downsample the Prudential facade with/without pre-filter → moiré on/off).

• sidebars: aliasing **without** Fourier (pixel = area average); aliasing **on purpose** (KinéCam moiré encodes camera motion).

• **ideal sinc** reconstruction (perfect frequency box ⇒ infinite spatial ripple) is unreachable; practical filters compromise (`fig-sinc-vs-practical`) — handed off to [[#Resampling and upsampling]].

• xref: built on [[#Fourier]]; reconstruction / upsampling lives in [[#Resampling and upsampling]].

reference the ML & deep-learning refresher ([[Refreshers#Machine learning and deep learning]]) up front; this chapter does **not** re-teach networks — it sets up *learning an operator from data* and the **data** that powers it. The concrete deep-network operators are the next chapter, [[Deep learning]].

these are the **deep-network** realizations of the learned-operator framing ([[Machine learning]]); architectures and training are the Refreshers' job — here we survey *what gets learned*.

This chapter is the **generative limit** of the two before it. *Machine learning* replaced a hand-designed prior with a **learned** one (L5.3); *Super-resolution* showed the prior is **not optional** and that a **denoiser is a universal prior** you can plug into any solver (L5.4, PnP/RED, and the punchline that the **score is a denoiser**). Here that same prior becomes something you can **sample from** — a generative model — and the canonical one, **diffusion**, is literally that denoiser run in a loop. We reference the ML / deep-learning refresher ([[Refreshers#Machine learning and deep learning]]) for U-Nets, transformers, and attention; this chapter does **not** re-teach networks — it surveys the *generative* idea and where it plugs back into the book's inverse-problem spine.

• the **histogram** — the distribution of pixel values — is the natural companion to point operations: this short chapter is how to *read* it, and how it can *drive* an automatic tone curve

• dead / stuck / hot pixels — what each reports, and why hot pixels grow with exposure time, heat, and ISO

• where they come from: manufacturing defects vs aging; dark current doubling every 6–8 °C

• finding them: factory defect map (ships in-camera / travels as DNG FixBadPixels) + dark-frame calibration (long-exposure NR, pixel mapping)

• fixing them: concealment by same-color interpolation, before demosaicking; the star-eater pitfall of over-aggressive suppression

• the part's claim made drivable: a raw editor is a programmable ISP exposed to the user

• two suggested experiments: Profile → Neutral (the beauty curve's contribution), and the clipping overlay on a bright sky (recoverable vs gone → bracketing, MULTIPLE EXPOSURE IMAGING)

• we **often need to compare two images** — fidelity, alignment, an ML training loss

• chapter intro: split out of [[Non-local means and BM3D]]; NL-means is the bilateral filter living in patch space, and the reason image self-similarity denoises so well.

• **the shift — filtering → optimization**: instead of one weighted-average *pass*, solve a **global least-squares** problem whose smoothness term uses the affinity as a weight. The same affinity, now inside an energy.

• **colorization** (the demo): the user scribbles a few colors; **propagate** them so that **neighboring pixels of similar luminance get similar color** — minimize $\sum_p (U_p-\sum_{q\in N(p)} a_{pq} U_q)^2$ over the chrominance $U$, with affinity $a_{pq}$ **large where $I_p\approx I_q$** (small luminance difference) — exactly the bilateral's range idea inside a quadratic energy

• **solve it** with the linear-systems machinery of the regression chapter — sparse, structured, solved by CG / multigrid (cross-ref [[Single-image computational photography#Blind deblurring]], [[Image formation as a linear system]])

• the **affinity matrix** here is the same object as the **matting Laplacian** and the graph-cut edge weights → forward-ref [[#Compositing, segmentation and matting]] (matting / segmentation = affinity again)

• also placeable under [[Single-image computational photography#Blind deblurring]] (the "Colorization as optimization?" stub) — keep the *technique* here, cross-ref from there

• **start here — Mertens *exposure fusion*, the gentlest pyramid method**: the cleanest way into this chapter is the technique that does **local tone mapping for free, without ever building an HDR image or choosing a tone curve**. Given a **bracketed stack** (or several renderings of one scene), score each pixel of each frame by simple **quality weights** — **contrast** (Laplacian magnitude, "is it sharp here?"), **saturation** (spread of the RGB channels, "is the color alive?"), and **well-exposedness** (closeness to mid-gray, "is it neither crushed nor blown?") — then keep, per pixel, the best-exposed frame. *(Mertens, Kautz & Van Reeth 2007.)* [`fig-exposure-fusion-mertens`]

• **the catch, and the fix — blend in the Laplacian pyramid**: a *per-pixel* weighted average of the frames **seams and halos** (the weights jump at boundaries between which frame "wins"). The fix is **pyramid blending** ([[Pyramids and wavelets]], Burt–Adelson): build a **Laplacian pyramid of each frame** and a **Gaussian pyramid of each (normalized) weight map**, blend the coefficients **level by level** (each frequency band fused at its own scale, with a correspondingly-blurred weight), then collapse. Because every band is feathered over a scale-appropriate transition, the join is invisible and no halos appear — the same reason pyramid blending hides a seam, now driven by exposure-quality weights.

• **why it opens the chapter**: it is the most intuitive instance of the chapter's one idea — *do the edge-aware / local-tone work **band-by-band in a Laplacian pyramid**, not pixel-by-pixel* — and it needs no range weight, no base/detail split, no radiance map. (Full HDR-capture context and where exposure fusion sits among merge-then-tonemap pipelines: [[Multiple exposure imaging]]; here it is the on-ramp to the pyramid machinery.)

• **from fusing a stack to reshaping one image — local Laplacian filters**: exposure fusion *blends several images* in the pyramid; **local Laplacian filters** (Paris et al. 2011) take the same "manipulate the Laplacian pyramid, halo-free" idea and apply it to a **single** image, reshaping its own bands by a pointwise remapping. Same home (the Laplacian pyramid), one input instead of a stack.

• **the odd one out — not an affinity method**: every other technique here weights *pairs* of pixels by similarity; local Laplacian instead works **band-by-band in a Laplacian pyramid** with a **pointwise remapping** — no range weight, no base/detail split (so none of the halos / gradient reversals those can produce). The multiscale cousin of the bilateral, by a different mechanism. Cross-ref [[Pyramids and wavelets]] (the pyramid it's built on) and [[Bilateral filtering]].

• **the mechanism**: to compute the output's Laplacian coefficient at a pixel and level, take the pixel's value $g$, apply a **remapping function** $r_g(\cdot)$ to the *whole* image (a smooth curve, centered on $g$, that decides what is "detail" vs "edge"), build a Laplacian pyramid of that remapped image, and read off **just this level's coefficient at this pixel**; collapse the resulting pyramid for the result.

• **the knob is the remapping $r$**: near $g$ (small amplitude = **detail**) it **compresses or amplifies** — controlling detail / local contrast; far from $g$ (large amplitude = **edge**) it stays the **identity** — so strong edges pass through untouched. One curve → detail enhancement, tone mapping, or inverse tone mapping, all **artifact-free**.

• **why it matters**: state-of-the-art **edge-aware detail manipulation and HDR tone mapping** with no halos and no gradient reversal — the failure modes that motivated the bilateral in the first place.

• **cost & the fast version**: the naive form rebuilds a pyramid per pixel-and-level (expensive); **Aubry et al. 2014** gives a fast approximation and shows the **unnormalized bilateral** is its single-scale relative (ties back to [[Bilateral filtering#Bilateral filter|the unnormalized bilateral]]).

• **the idea**: like the joint bilateral (filter $I$, take structure from a **guide** $J$) but the output is a **local linear transform of the guide**, $I^{\mathrm{out}}=a_k J + b_k$ fit per window by **least squares** — so it preserves the guide's edges by construction

• **why prefer it**: **$O(N)$**, **no range-quantization artifacts** (the gradient reversal the bilateral can show), and **differentiable** → drops into learning pipelines

• **uses**: edge-preserving smoothing; detail enhancement; **flash / no-flash**; **matting / feathering**; **dehazing** (guide = the hazy image — xref He, Sun & Tang 2009 dark-channel, [[#Dehazing as a prior-driven inverse problem]]); depth upsampling

• the affinity is now **implicit** in the local linear fit — same family, different mechanism; a fitting end-point for the chapter's through-line (bilateral → grid → learned → regression → optimization → guided, all *affinity*)

• **the view**: treat denoising / upsampling as **local regression** — fit a smooth function to nearby pixels, weighted by a kernel

• **steering / adaptive kernels**: shape the kernel to the **local image structure** — estimate the local gradient, then **stretch the kernel *along* edges and squeeze it *across* them** (an anisotropic, data-dependent weight) so it averages along the edge, not across it

• so LARK is the **regression flavor** of the same affinity idea: the kernel geometry is *derived from* local structure, generalizing the bilateral's scalar range weight to an oriented one

• also a **descriptor**: LARK signatures are used for detection / matching (Seo & Milanfar) — note in passing, not developed

• **the third leg of EDGES MATTER** (the part's framing payoff): Poisson **reconstructs from edges**, bilateral **smooths except across edges**, seam optimization **cuts along edges**. State it as the dual of edge preservation: edge-preserving filtering works *hard to avoid* edges; seam methods *go looking for them* — a transition is invisible exactly where the image is already changing, so we hide the cut **in the edges / busy texture** and keep it **out of smooth regions** where a seam would scream. (Callback to the part intro's "respect the edges.")

• the **unifying machinery**: all three sub-families are **discrete optimization on a graph or grid** — a 1-D least-cost path (**DP**), a 2-D globally-optimal boundary (**graph cut / min-cut**), or a spectral partition (**normalized cuts**). What changes between methods is **the energy / affinity**, not the optimizer.

💡 **Big lesson:** *many image edits are **discrete optimization on a graph** — dynamic programming, min-cut/max-flow, or spectral relaxation — and the **energy (affinity) you choose is the whole game**.* The optimizer is off-the-shelf; **what** you penalize (gradient magnitude, label disagreement across edges, region/boundary cost) is the design. This is the **L4.21 affinity** principle (Bilateral) turned into a *cut*: there the affinity said "average these together"; here it says "**don't cut here**." *(Register as new lesson **L6.4 · Image edits as discrete optimization on a graph (the energy is the design)**, first appearance here; "also relevant": Poisson/photomontage, compositing/segmentation/matting, texture synthesis, stereo/MRF labeling. Full box at this section's head; one-line callback to L4.)*

💡 **Big lesson (L16 · Prefilter before you downsample):** a warp that *shrinks* the image (minifies) is throwing samples away — and if you just decimate (keep every $N$th source pixel, or sample a narrow reconstruction kernel) the detail too fine for the new grid **folds down into bold false moiré**, not into nothing. The fix is to **area-average first**: widen the resampling kernel in proportion to the downscale factor $s$ (so it gathers over the whole footprint each output pixel now covers) and renormalize — that low-pass blur is the prefilter. Enlarging needs *no* prefilter (clamp the kernel width to 1); shrinking *always* does. (→ see Big lesson **L16**, first placed in BASIC → Resampling; the frequency-domain *why* is **L4.8** — aliasing folds frequencies above the new Nyquist. The same lesson governs every geometric resample in this part: rectification corners that shrink, stabilization crops, optical-flow rendering.)

⚠️ **Candidate new part-spine lesson (flag only — not registered).** Almost every algorithm in this part follows one shape: **establish a correspondence / coordinate map between two images (or an image and a target frame), then transport pixels along it.** Warping makes the map explicit and applies it; perspective rectification *solves* the map (a homography) from constraints; panorama stitching *estimates* the map from features; morphing *interpolates between* two maps; optical flow *estimates a dense map* from brightness constancy; stabilization *smooths a map over time*; frame interpolation *renders along* a map. A lesson like **"find the coordinate map, then move the pixels"** would unify the part the way L5.4 unifies the inverse-problem part. Worth registering as a part-spine lesson for MOTION/WARPING — *noted here, deliberately not registered* (registry-first convention: register in [[Big Lessons]] before placing).

💡 **Big lesson (L16, anisotropic):** the prefilter-on-minify discipline (**L16**) is *direction-dependent* the moment the warp stops being a similarity. A general warp compresses the source more along one axis than another, so the right prefilter is not a round blur but an **oriented ellipse** — average a lot along the compressed direction, barely at all across it. Isotropic filtering must then either alias the compressed axis or blur the sharp one; anisotropic filtering (EWA and its approximations) does both jobs at once. Same lesson, one dimension richer.

💡 **Big lesson (recurrence — *align before you blend*):** averaging two images only makes sense once their features sit at the *same place*. A plain cross-dissolve blends colors at fixed coordinates, so any misalignment **double-exposes into a ghost**; the cure is to **warp into correspondence first, then average** — the same move as panorama de-ghosting and multi-frame super-resolution's sub-pixel alignment. Morphing is this lesson taken to its limit: not just align-then-average, but interpolate the *alignment itself* over $t$. (Companion to the *capture-then-decide / align-then-merge* family; the dense automatic aligner is [[Optical flow]].)