Every big lesson in the book, organized by part and chapter (155 total). Each links to where it is first developed; on the local edit server the ⧉ opens the lesson's own source draft so an edit here lands on the original.
Part 1 — Intro
1.1 From digital to computational photography
L1.1a useful technique is fast, predictable, and controllablewhere →⧉
A useful method meets its latency and energy budget, degrades acceptably when its assumptions fail, and exposes controls that correspond to photographic decisions. Establish correctness before optimizing; measure both image quality and runtime on representative inputs.
Light, and the physical color it carries, is described by a spectral distribution: the power it carries at each wavelength, a function $S(\lambda)$. One wavelength is a single pure color; real light is a weighted mix of all of them. Everything downstream (how a prism fans light out, how a surface reflects it, how a sensor or an eye turns it into a perceived color) starts from this per-wavelength power.
L2.2brightness is a quadratic quantity of the fieldwhere →⧉
What we call brightness or intensity is the power of the electromagnetic field, a quadratic quantity of the field. We never record the field itself, only its time-averaged square, so brightness is always non-negative and blind to the field's sign and phase. And because it is quadratic rather than linear in the field, two light contributions do not simply add in brightness: squaring their summed fields leaves a cross term, and whether that term survives or averages away is what coherence decides.
Coherence decides which quantity of light is linear. Coherent light superposes amplitudes, so its cross terms survive as interference; incoherent light superposes intensities, because the random phases average the cross terms away. No source is fully one or the other, only more or less coherent at a given scale of time and space. Ray optics is what emerges in the incoherent, short-wavelength corner of this space: interference washes out, intensities add along paths, and light travels as independent rays of energy. Most photography involves incoherent light, which means that it is linear in intensity.
At a boundary between two media, light bends by $n_1\sin\theta_1 = n_2\sin\theta_2$, toward the normal entering a denser medium, away from it on the way out. This single rule is the seed of all lens optics: every focal length, every ray trace, the thin-lens equation, and every aberration is Snell's law applied at curved glass surfaces. Pair it with the fact that the index $n$ depends on wavelength (the next lesson) and you also get dispersion. Almost everything optical in this book traces back to it.
The refractive index is not a constant: $n(\lambda)$ depends on wavelength: in ordinary glass it rises toward the blue, so blue bends more than red. Because of this you cannot route every wavelength the same way with a single refraction: a prism fans white light into a spectrum, a raindrop makes a rainbow, and a simple lens focuses blue and red at slightly different distances: chromatic aberration. Much of the craft of lens design (combining crown and flint glass into achromatic and apochromatic groups) exists to fight this one fact. Dispersion is not a defect of cheap glass; it is built into how matter and light interact.
Much of imaging is multiplicative, not additive. The light reaching the eye is illumination times reflectance; contrast is a ratio of intensities; a shadow darkens a region by a factor, not by subtracting a constant. Because the structure is multiplicative, the natural tools are multiply, divide, and log, not just add. This is why, again and again, we will work with ratios and with logarithmic encodings rather than raw linear values. It is also why white balance is hard: to recover an object's true color we must divide out the unknown illuminant $S(\lambda)$ from the product $C(\lambda)$, undoing a multiplication we never observed directly. (The additive counterpoint, light from different sources adding, and what it means for choosing an encoding, comes in Color technology.)
L2.7diffuse means the same color from every directionwhere →⧉
A diffuse (Lambertian) surface looks the same brightness and color from every viewing direction, its appearance does not depend on where you stand, only on where the light is. This is an idealization (real surfaces add a view-dependent specular sheen), but it is one a huge amount of computer vision quietly assumes, because it lets a scene point have a single color that can be matched across views. Stereo, multi-view and photometric stereo, optical flow's brightness-constancy, and any "just average the color at this point" reasoning all lean on Lambertian reflectance, and shiny, view-dependent highlights are exactly where they break.
L2.8radiance is conserved along a ray, so surfaces do not dim with distancewhere →⧉
A ray carries the same radiance however far it travels through empty space. A white wall does not get darker as you walk away from it, it only subtends a smaller angle, so fewer of its rays land in each pixel. The familiar inverse-square falloff is therefore a statement about a source's solid angle collapsing as $1/r^2$, not about the ray losing energy. Keeping the two apart is what makes radiometry, exposure, the f-number, and HDR merging come out right.
The light a surface receives per unit area is thinned by the cosine of the incidence angle: a beam striking at angle $\theta$ from the normal spreads its power over a footprint $1/\cos\theta$ larger, so irradiance $\propto \cos\theta$ (Lambert's cosine law). This $\cos\theta$ is not a fudge factor, it is projected area, and it reappears everywhere: it is why a matte sphere fades toward its edge ($\mathbf n\!\cdot\!\mathbf l$ shading), why the poles are colder than the equator, why lenses vignette toward the corners ($\cos^4$), and it sits inside every radiometric integral over the hemisphere.
L2.10the smaller the aperture, the more diffraction blurswhere →⧉
Counter to ray-optics intuition, a smaller aperture spreads light more: the diffraction blur (the Airy disk) grows as $\sim\!\lambda N$ with f-number, so stopping down past a point softens the image instead of sharpening it. The flip side is the rule behind every instrument that must resolve fine detail: only a large aperture can diffract little and resolve finely, which is exactly why microscopes and telescopes are built around large lenses and mirrors, and why a phone camera's tiny aperture hits a hard resolution wall it cannot pass. It is the optical cousin of "you can't sample your way past Nyquist."
2.2 Pinhole image formation and linear perspective
L2.11image formation is choosing which rays reach each pointwhere →⧉
Forming an image is, at bottom, controlling which ray (or rays) of light contribute to each point of the image. A bare sensor fails because every scene point sends light to every pixel. The sum is a flat gray. A pinhole admits one ray per point (sharp, but dim); a lens gathers a whole cone of rays from one scene point and bends them back to one image point (bright, but only for one focal plane); an open hole with no focusing mixes many rays and blurs. Every imaging idea in this book (focus and depth of field, coded apertures, light fields, even a CT scanner) is a different answer to the one question: which rays land here?
L2.12field of view depends on the sensor sizewhere →⧉
For a pinhole a fixed distance $f$ from the sensor, the field of view is set by the sensor size: $\text{FOV}=2\arctan(\text{sensor}/2f)$. Hold $f$ fixed and slide a larger sensor behind the same pinhole and you take in more of the world; a smaller sensor crops to a narrower view. So "how wide a shot is" is never a property of the pinhole (or, next chapter, the lens) alone: it is the pinhole-to-sensor distance and the sensor size, together. This is the geometric root of the crop factor and of "35 mm-equivalent" focal lengths (next chapter), and the reason the same lens is wide on a big sensor and tele on a small one.
From a pure geometric-projection standpoint, cropping an image and zooming in (lengthening the focal length) are one and the same operation. Both leave the perspective projection (the divide-by-$Z$ onto the normalized plane) completely untouched and change only the pixel stage $K$: a crop keeps a smaller patch of the projected image and rescales it to fill the frame, which is exactly what a larger $f_x, f_y$ does. So a 2× crop and a 2× zoom give the identical framing and identical perspective, differing only in resolution: the crop simply has fewer pixels to show for it. This is why "digital zoom" is geometrically free but costs detail, and why the crop factor between sensor sizes acts as a focal-length multiplier.
L2.14linear perspective is a divide by depthwhere →⧉
With the camera at the origin looking down $z$, a scene point $(X,Y,Z)$ lands at $(f X/Z,\; f Y/Z)$: the whole of perspective is dividing by depth. That one division produces everything we call perspective: farther things appear smaller (double the depth, halve the size), parallel lines converge to vanishing points, and the picture gains its sense of space. It is also why depth is the hard thing to recover: the division throws $Z$ away, and a single image cannot tell a small near object from a large far one. Every stereo, structure-from-motion, and depth-estimation method in later parts is, in effect, trying to undo this divide.
L2.15a general camera is a matrix, then a dividewhere →⧉
A camera in any position and orientation projects a 3D point in two moves: a rigid transform that brings the world into the camera's frame (because moving the camera equals moving the world by the inverse), followed by the perspective divide. In homogeneous coordinates the two compose into a single $3\times4$ matrix $P$, applied to the homogeneous point $(X,Y,Z,1)$, and the pixel is recovered by dividing by the last coordinate. So any pinhole view, however the camera is placed, is "one matrix multiply, then one division." That divide is the only nonlinear step in all of projection, the generalization of L2.14's divide-by-depth to an arbitrary viewpoint.
L2.16zooming in is not the same as moving closerwhere →⧉
You can make a subject fill the same fraction of the frame either by zooming in or by stepping closer, and at that one depth the two give the identical magnification. But magnification falls off with distance, so everything at other depths behaves differently. Changing focal length rescales the whole projected image by a single factor (it is the pixel stage $K$, the "crop = zoom" lesson again), preserving every near-far size ratio; changing your distance rescales near and far by different factors, because each object's magnification depends on its own depth. Step close and the near looms while the far recedes; zoom from afar and the background stacks up "compressed" behind the subject. So relative near-far size (what we call perspective or compression) is set by viewpoint (distance), not by the lens; the focal length only chooses how much of that view to crop. The dolly zoom below is this lesson made vivid.
L2.17focal length does not specify the field of view without a sensor size, hence the 35 mm equivalentwhere →⧉
The field-of-view-depends-on-sensor-size lesson (L2.12) carries straight over to a real lens: for a fixed focal length $f$, the field of view still follows $\text{FOV}=2\arctan(\text{sensor}/2f)$, so the same lens is wide-angle on a big sensor and telephoto on a small one: the smaller sensor simply crops the projected image into a narrower view. A focal length in millimeters is a real property of the lens, but on its own it does not specify the field of view: you have to name the sensor too. That is why focal lengths are quoted as a 35 mm- (full-frame-) equivalent: the focal length that would frame the same way on a $24\times36$ mm sensor. Concretely, a phone's main camera has an actual focal length of only about 6 mm, but on its tiny (~1/1.3-inch) sensor it frames like a 24–26 mm lens on full frame, so it is sold as "26 mm equivalent." The conversion is just the crop factor, full-frame diagonal ÷ sensor diagonal (≈ 7× for that phone), multiplying the real focal length.
2.3 Lens image formation
L2.18a lens makes a large set of ray from an in-focus point reconverge to one image pointwhere →⧉
Where a pinhole keeps only one ray, a lens gathers a whole cone of rays leaving a scene point and bends them back to a single image point, but only for points at the focus distance. That is the lens's compromise: vastly more light (a bright image) in exchange for a single plane of sharp focus, with everything nearer or farther spread into a blur disk (the circle of confusion). The aperture sets how wide that cone is, and hence the depth of field. Every later idea about focus, bokeh, refocusing, and light fields is a variation on "which cone of rays reconverges where."
L2.19when a quantity is small, linearize: keep the first Taylor termwhere →⧉
What we just did to Snell's law is one of the most powerful moves in all of physics and engineering, and the book will reach for it again and again. Replace a messy nonlinear function by its tangent line near the operating point, keep only the first-order (linear) term of its Taylor expansion, and it is an excellent approximation as long as the relevant quantity stays small. Here the small quantity is the angle, $\sin\theta\approx\theta$, and the reward is the entire clean, linear machinery of paraxial optics. The same reflex linearizes brightness change into the optical-flow constraint (a chapter away), drives the Gauss–Newton steps of lens optimization and bundle adjustment, and propagates small noise through a pipeline. Two disciplines keep it sound. Know when it breaks: the approximation is only as good as the terms you dropped are negligible, so always ask "how small is small?" And keep the next term in your pocket: restoring it is not a footnote but a whole theory: for optics that next term is the cubic $-\theta^3/6$, and restoring it is the study of aberrations (next big lesson, and the OPTICS part).
L2.20thin-lens optics is a linear (first-order) approximation of Snell's law when angles are small (the paraxial regimewhere →⧉
The clean thin-lens equation $1/f = 1/u + 1/v$ is not a separate law: it is Snell's law L2.4 linearized, kept to first order under the small-angle approximation $\sin\theta\approx\theta$, the regime known as paraxial (rays near the axis, gentle angles, one wavelength). Because it keeps only the linear term, it tells you exactly when the idealization holds and exactly how it breaks: restore the next term $\sin\theta\approx\theta-\theta^3/6$ and the Seidel aberrations appear; let $n$ vary with wavelength and chromatic aberration appears. "Thin lens," "first-order optics," "a linear (paraxial) lens," and "a lens with no aberrations" are all the same approximation, and the OPTICS part is the story of paying back what it borrowed.
Parallel rays through a thin lens converge to a point at the focal length, and rays going through the center of a thin lens do not deviate. The first rule is what focal length means (where a far subject's parallel bundle meets) and carries the lens's focusing power; the second carries its perspective, identical to the pinhole's. Between these two construction rules and similar triangles, you can derive all of thin-lens optics.
L2.22the focus equation is symmetric: object and image are interchangeable conjugateswhere →⧉
The thin-lens equation $\tfrac{1}{u}+\tfrac{1}{v}=\tfrac{1}{f}$ is symmetric in $u$ and $v$: swap them and it is unchanged, so object and image are interchangeable. Send the light back the other way and the rays retrace exactly the same path (the reversibility of light, made arithmetic). Two consequences fall straight out: an object at infinity images at $v=f$, which is what focal length means, and focusing a camera is nothing but moving $v$ until the chosen object's conjugate lands on the sensor.
L2.23the simplest way to focus is to move the sensor: full (unit) focusingwhere →⧉
The most basic way to focus follows straight from the equation: move the whole lens relative to the sensor, changing the lens-to-sensor distance $v$ until the chosen subject's conjugate lands on the sensor. This is full or unit focusing, the entire optical unit sliding with its focal length fixed. A bellows, the helicoid focus ring on a simple prime, and a bare enlarger lens all work this way, and it is the mental model to keep. It is only the simplest option, though. A compound lens has many elements, and moving just some of them (front-cell, internal, or rear focusing, sometimes with floating groups) can focus while extending the barrel less, holding aberrations in check across the focus range, or even nudging the effective focal length. Those techniques, and why designers reach for them, come in the OPTICS part's focus section.
L2.24magnification is the ratio of projected size to real-world sizewhere →⧉
Magnification is the size of a subject's image divided by the subject's actual size in world units (meters): $m = -v/u$ is exactly that ratio (image height over object height). Photographers quote it as a reproduction ratio: 1:1 ("life size") means the image on the sensor is as large as the subject itself; 1:2 means half life size. What fraction of the frame a subject then occupies depends on the sensor size: the same magnification fills more of a small sensor than a large one. On a full-frame ($24\times36$ mm) camera a $1{:}2$ reproduction images a subject about $48\times72$ mm to exactly fill the frame; put that same subject on a smaller sensor at the same magnification and it overflows. So magnification is a property of the optics-and-distance alone, while "how big it looks in the picture" also needs the sensor size.
L2.25the f-number is the aperture diameter as a fraction of the focal lengthwhere →⧉
$N = f/D$: the f-number tells you the width of the opening measured in focal lengths, not in millimeters. $f/4$ means the opening is one quarter of the focal length across. Writing the aperture this way is what makes it a portable brightness setting: because the cone of light converging on each sensor point has half-angle $\tan\theta = 1/(2N)$, the ratio alone sets the image brightness, so the same f-number gives the same exposure on any lens, whatever its focal length or physical size. The denominator is why a small number is a big opening, and the area-goes-as-diameter-squared law ($\text{light}\propto 1/N^2$) is why the stop series climbs by $\sqrt2$.
2.4 Image measurements as integrals over the plenoptic function
No pixel measures "the" value at a point in space, time, or color: it reports an integral over everything the photosite cannot tell apart. Four axes appear explicitly above: the spatial footprint (the pixel's area, plus the optical blur smeared across it), the exposure time (a window, not an instant), the spectrum (wavelength, weighted by the channel's spectral sensitivity), and the solid angle of rays the aperture admits; a fifth, polarization, is usually lumped in too. Naming all the axes at once is what makes motion blur, defocus, color, vignetting, and aliasing the same phenomenon (a finite integral standing in for an idealized point sample), and the width of the integral along each axis is a knob the rest of the book spends its time controlling.
2.5 Depth of field
L2.27depth of field is set by a maximum acceptable circle of confusionwhere →⧉
A defocused point never images as a point but as a blur disk: the circle of confusion. The depth of field is then defined as the range of distances over which that disk stays smaller than a chosen maximum $c$: inside the depth of field every point is blurred less than $c$, outside it more. The operative word is acceptable: $c$ is not a constant of nature but a judgement call that depends on the target application, the viewing distance, and the sensor (or print) resolution. A billboard read from across the street tolerates a huge $c$; a 100% crop on a 4K monitor almost none. Change the standard and the depth of field changes with it, even though the optics did not.
L2.28depth of field scales linearly with f-number, quadratically with distance, inverse-quadratically with focal lengthwhere →⧉
Away from the macro range (subject much closer than the hyperfocal distance), the total depth of field follows a clean scaling, $\text{DoF}\approx \dfrac{2\,N\,c\,s^2}{f^2}$: it grows linearly with the f-number $N$ (stop down for more), quadratically with the focusing distance $s$ (step back and it balloons), and inversely with the square of the focal length $f$ (wider for more, longer for less). These three exponents explain at a glance why a phone (tiny $f$) keeps almost everything sharp, why close focus annihilates depth of field, and why "I need more in focus" always reduces to stop down, back up, or go wider.
L2.29many approximations break in the macro regimewhere →⧉
The convenient first-order results of this chapter (the depth-of-field scaling just above, the framing shortcut "magnification $\approx f/\text{distance}$", even the bare f-number as a stand-in for image brightness) all quietly assume the subject is far compared with the focal length. Up close (macro, at magnifications near or past 1:1) they break. The pupil magnification stops being 1, the effective f-number grows as $N(1+m)$ so you lose real light, the working distance shrinks toward the front element, and the depth of field collapses to millimeters. In the macro regime you must drop the shortcuts and return to the exact conjugate geometry (and, often, to focus stacking). It is a good reflex to ask of any optics approximation: does this still hold when the subject is this close?
L2.30focus at the hyperfocal distance and everything from half it to infinity is sharpwhere →⧉
There is a single focus setting that extracts the maximum depth of field from a lens: the hyperfocal distance $H = f^2/(N c)$. Focus there and the far limit runs all the way to infinity while the near limit sits at exactly $H/2$, so the entire scene from $H/2$ to the horizon falls inside the acceptable circle of confusion. Focusing at infinity instead throws away the whole sharp zone between $H/2$ and $H$ for nothing. It is the landscape-and-street photographer's standard move for front-to-back sharpness, and the basis of zone focusing (pre-set the focus, shoot without refocusing).
L2.31near the subject, depth of field follows magnification; in the far background, the long lens winswhere →⧉
Two different questions get confused. Near the plane of focus, at equal magnification and equal f-number, the depth of field is essentially independent of focal length: stepping back to hold the framing gives back exactly what the longer lens took. In the distant background, the longer lens does defocus more, because what blurs a far object is the physical aperture diameter $D=f/N$, which grows with focal length at a fixed f-number. So "telephotos have shallow depth of field" is wrong about the subject and right about the background.
L2.32in image space, defocus blur shrinks linearly with the sensorwhere →⧉
Hold the framing and the f-number fixed and scale the camera down: the defocus blur of the background, as a fraction of the frame, shrinks linearly with the sensor size, because it tracks the lens's physical aperture diameter $D=f/N$ and $f$ shrinks with the sensor. So shallow depth of field is a privilege of physically large apertures (a big sensor with a big lens) and a small sensor cannot fake it optically. This is why phones have deep depth of field and must compute fake background blur (portrait mode, a later chapter): the tiny sensor shrinks every blur disk toward nothing. It is the image-space twin of "small sensor → more depth of field" above.
2.6 Motion blur
L2.33rotation blur is distance-independent (yaw/pitch scale with focal length, roll with image position); translation and subject blur scale as 1/distancewhere →⧉
Camera rotation (yaw/pitch) smears the image by $b_{\text{px}} \approx f_{\text{px}}\,\theta$, the same for every object regardless of depth. Camera translation and subject motion smear it by $b_{\text{px}} \approx f_{\text{px}}\,(\Delta \text{ or } v\Delta t)/D$, which shrinks with distance. So for ordinary distances rotation dominates, and only up close (macro) does translation take over. The focal-length dependence divides the same way: yaw and pitch blur scale with $f_{\text{px}}$, so a longer lens (higher magnification) turns the same angular wobble into proportionally more blur (the basis of the $1/f$ hand-holding rule), while roll blur $b \approx r\,\theta_{\text{roll}}$ is fixed by image position $r$, not by focal length. Magnification amplifies yaw and pitch but never roll, which is why yaw and pitch usually dominate the shake blur, the more so the longer the lens. And because roll and perspective make $b$ vary across the frame, camera-shake blur is spatially non-uniform: a single blur kernel cannot describe or undo it.
L2.34the exposure ends with whichever is shorter, the shutter or the lightwhere →⧉
With the ambient dark, a burst lasting microseconds freezes motion however long the shutter stays open, because the sensor only integrates while light arrives. Illumination is a second clock on the time axis, independent of the mechanical one and far faster than it. The bill comes as total light: a shorter burst carries proportionally less of it, which is why freezing faster costs aperture, ISO, or distance. Everything in computational illumination, from stroboscopy to Edgerton's microsecond sparks, is written on this clock.
2.7 Sensors: photosites, CCD vs CMOS
L2.35sensors convert photons to electrons linearlywhere →⧉
A photosite is at heart a photon counter: each absorbed photon frees a charge carrier, an electron or equivalently a hole (a missing electron), and the well accumulates them in direct proportion to the light that arrives. So the raw signal is linear in scene radiance, before any gamma or tone curve. This linearity is not a detail; it is what makes raw arithmetic trustworthy: HDR merging adds exposures, white balance is a per-channel multiply, flat-field correction is a divide, and all three are valid only because the numbers are proportional to light. Display encoding (gamma) deliberately hides this linearity for the eye, so computation should undo it and work in linear light.
L2.36sensor size is one dial behind field of view, depth of field, and light; the equivalences scale linearly with it, the light with its areawhere →⧉
Almost every camera trade-off scales with the sensor's linear size, captured in the crop factor $c$ (full-frame diagonal $\div$ the sensor's). Three consequences, each with a different scaling: - Field of view: to match a framing, the equivalent focal length scales linearly, $f_{\text{eq}} = f \times c$ (L2.12, L2.17): a phone's $6$ mm lens frames like $\sim\!26$ mm. - Depth of field: at fixed framing and f-number the physical aperture diameter, and so the background blur, scale linearly with sensor size, so the equivalent f-number is $N_{\text{eq}} = N \times c$ L2.32: a phone at $f/1.8$ has full-frame's $f/12$ depth of field. Small sensors give deep, near-universal focus and little bokeh. - Light and noise: at equal framing and f-number the total light collected scales with sensor area ($\propto c^{2}$), so the photon count does too; photon-noise-limited SNR and dynamic range then scale with the linear dimension ($\sqrt{\text{area}}$, $\propto 1/c$) L2.38. This total-light advantage, not pixel count, is why a big sensor looks cleaner. So the bookkeeping is: equivalent focal length and equivalent f-number are linear in sensor size; total light gathered is quadratic (its area); the SNR and dynamic-range benefit go as the linear dimension (the square root of area). One dial, three different exponents.
2.8 Noise, signal-to-noise ratio and dynamic range
L2.37variance is the mathematically meaningful quantity; standard deviation is the human-readable onewhere →⧉
Two numbers describe noise spread, and both exist for a reason. Variance (the mean squared deviation) is the quantity the mathematics wants: it is additive for independent sources, exactly what we just used, and the basis of the $1/\sqrt N$ averaging law in Slopendium/Denoising; it is the second moment that falls out of every Gaussian and least-squares derivation, and it composes cleanly under linear operations. But its units are squared (squared gray-levels, squared photons), so it means nothing to the eye. The standard deviation $\sigma$ (its square root) is the one people should read: it is back in the original units, directly comparable to the signal: the "$\sigma$" of error bars and the grain you actually see. Rule of thumb: derive and combine in variance, report in standard deviation.
L2.38noise is fundamental: photon shot noise sets the floorwhere →⧉
Even a flawless sensor is noisy, because light itself is grainy: photons arrive as discrete, randomly-timed events, so the count in any fixed interval follows Poisson statistics ($\sigma=\sqrt N$), and no electronics can remove it: the randomness is in the light, not the chip. The only true cure is collecting more photons: a bigger pixel, a wider aperture, a longer exposure, or averaging many frames. Raising ISO is not a cure: it is gain applied after capture, amplifying signal and shot noise together. This irreducible floor is why bigger sensors win, why shadows are noisy, and what every denoiser later in the book is ultimately fighting.
L2.39know the noise sources, and when each one dominateswhere →⧉
Sensor noise is a sum of distinct sources whose variances add, but each rules a different regime, so diagnosing a noisy image means asking which one you are seeing. Photon shot noise ($\propto\sqrt{\text{signal}}$) dominates the midtones and highlights, and any well-lit shot. Read noise is signal-independent and dominates the deep shadows and short exposures, setting the floor. Thermal / dark-current noise grows with exposure time and temperature: the long-exposure and astrophotography enemy. Quantization matters only with too few bits L2.47. And fixed-pattern noise / PRNU (static per-pixel gain and offset variation) reads as a structured pattern and is removed by calibration (dark and flat fields). Match the symptom to the source before reaching for a fix.
L2.40aggregating measurements keeps the shot noise but multiplies the read noisewhere →⧉
Add the readings of $k$ small pixels and you get the same Poisson (shot) noise as one big super-pixel that caught the same photons, but you pay the read noise $k$ times, once per read-out, instead of once. So the only fundamental advantage of a bigger pixel (or one longer exposure) over many smaller ones is the read noise, not the photon statistics, as long as no light is lost in the gaps. The same holds for aggregating along any dimension: summing $k$ short exposures, or $k$ burst frames, keeps the shot noise of the total light but charges read noise $k$ times. This is precisely why charge-domain binning (combine, then read once) beats digital summation (read $k$ times, then add) in the dark.
L2.41bright pixels are *noisier* (more absolute noise) but have *better SNR*where →⧉
Shot noise is Poisson, so its variance grows linearly with the signal ($\sigma^2 \propto N$) and its standard deviation grows only as the square root ($\sigma \propto \sqrt{N}$). Two consequences pull in opposite directions: the absolute noise is larger in highlights than in shadows (a 10,000-photon highlight has $\sigma=100$; a 100-photon shadow has $\sigma=10$), yet the SNR $= N/\sqrt{N} = \sqrt{N}$ is higher in the highlights: the signal outgrows the noise. So bright areas are noisier in absolute terms but cleaner to the eye; the shadows have the worst SNR (a figure, a figure). This is why exposing to the right and brighter scenes look smoother, and why we gamma/log-encode so equal ratios get equal steps.
L2.42in a *linear* image, noise variance is mostly an *affine* function of brightness (ignoring clipping).where →⧉
The variances add, and two terms dominate: shot noise contributes an amount proportional to the signal (Poisson, variance = mean, scaled by the gain) and read noise a constant floor. So the total is variance ≈ gain·signal + read², a straight line in brightness (a figure). The slope is the photon gain (it steepens with ISO, which amplifies the signal and its shot noise together); the intercept is the read-noise floor. We can measure it without any model: take an aligned burst of a static scene and, per pixel, plot the variance across frames against the mean. This affine law is a statement about the variance and is essentially exact for the two dominant sources; it says nothing yet about the shape of the distribution. Where the electron count is large, a single pixel's value over many frames does look ≈ Gaussian (a figure), which licenses modeling per-pixel noise as additive Gaussian in the midtones and highlights. That bell is only an approximation, though: the shot term is really Poisson (variance = mean, but discrete, skewed, and non-negative), and in the deep shadows, where only a handful of electrons land, it stops looking Gaussian at all. The additive-Gaussian model is a convenience of the well-exposed regime, not a law of noise in general (we pin down where it breaks below).
L2.43noise is *clipped* at black and white, so near the extremes it is *not* zero-meanwhere →⧉
A recorded value is clamped to $[0,\text{max}]$. Near black the negative half of the noise is cut off and frames pile up at 0; near white the positive half is. So at the extremes the noise distribution is asymmetric, with its mean pushed inward (a figure). The consequence for denoising: a naive average or smoothing returns that biased mean, so it makes shadows come out too bright and highlights too dark: a bias every denoiser must correct for (we return to it in Denoising, BASIC).
L2.44the highlight clip is *soft*: over-range pixels do not all read the maximumwhere →⧉
The top rail is not a hard wall. Read noise is added after the charge saturates at the well (clip 1) and before the ADC clamps (clip 2), so a pixel pinned at full well is digitized to $g\,N_{\text{full}}+B$ plus a zero-mean Gaussian excursion: about half of all saturated pixels read strictly below $\mathrm{DN}_{\max}$, by up to a read-noise's worth. It makes no difference how far over-range the pixel truly is: a highlight ten times too bright still only fills the well to $N_{\text{full}}$, gets the same read noise, and lands below the maximum half the time. So a measured value just under the ceiling does not prove a pixel was unsaturated, and a patch of genuinely clipped highlights reads a little spread of values, not one flat number. Anything that must know what saturated (HDR-merge pixel rejection, highlight reconstruction, clipping warnings) has to threshold a few $\sigma_r$ below $\mathrm{DN}_{\max}$, never exactly at it.
L2.45the full noise model is Poisson, then clip, then Gaussian, then clipwhere →⧉
A single measured value is built in four stages, in the order the sensor applies them: (1) Poisson photon shot noise, whose variance equals its mean, the irreducible floor; (2) saturation at the well, a hard clip of the electron count at the full-well capacity; (3) additive Gaussian read noise, the electronics' contribution, applied after the charge is held; and (4) the ADC clip to $[0,\mathrm{DN}_{\max}]$, after gain and the black-level pedestal. Do it in electrons, the only unit in which shot noise is exactly Poisson. The tidy affine Gaussian ($\sigma^2=a\,I+b$) is just the well-exposed limit of this recipe with the Poisson shape and both clips dropped, which is why it matches in the midtones and parts from the truth in the deep shadows (skewed, floored) and at saturation (a soft, spread-out ceiling). The order matters: because read noise lands between the two clips, saturated highlights do not pile up as a clean spike (see the two clipping lessons above). Get it right and you reproduce real noise; get it wrong and your simulator, training data, or denoiser is fitting a sensor that does not exist.
L2.46dynamic range = full-well capacity ÷ noise floorwhere →⧉
A single exposure records from a top (the photosite's full-well capacity, where it saturates and clips to white) down to a bottom (the noise floor, the read noise that drowns the shadows). Their ratio is the dynamic range, quoted in stops. That floor is not absolute: it sits wherever the SNR drops below what you deem acceptable, so, just as with depth of field and its circle of confusion, the number depends on the quality criterion you pick (SNR = 1 gives the generous engineering range; a stricter SNR gives the narrower usable one). You widen it with a bigger well (larger photosites, why a full-frame sensor out-ranges a phone) or a lower floor (cooling, lower read noise), and you beat it altogether by merging multiple exposures (HDR). This is the capture-side companion to the quantization lesson below: what bounds a single shot is this range, full-well over floor, not the number of bits.
L2.47quantization is rarely the real problemwhere →⧉
With enough bits and a sane encoding, what limits image quality is noise and dynamic range (the two quantities of this section), not the number of levels the analog-to-digital converter (ADC) offers. The noise floor and the full well bound what you can record; the quantization step, once it sits below the noise, is invisible. (Indeed, a little noise dithers the quantization, hiding it further.) Levels matter in one place: banding in the deep shadows of a linearly-encoded image, too few codes where the eye, working in ratios, is most sensitive, and that is exactly the problem gamma encoding exists to solve, by spending code levels perceptually (Color space 101). So "rarely" is not "never." We meet this lesson again in BASIC → Image representation → Float vs 8-bit, where floats with headroom are the sane default.
Part 3 — Visual perception and color
3.2 Perceptual color and trichromatic vision
L3.1a cone response is a dot product of the spectrum with the cone's sensitivitywhere →⧉
Each cone excitation is an integral of spectral density times the appropriate sensitivity. A sampled implementation must include wavelength-bin widths (or absorb them into the matrix). Three excitations summarize photopic color matching for a fixed observer and viewing condition; they do not recover the spectrum.
L3.2metamers: different spectra, same cone responsewhere →⧉
Different spectra can yield the same cone excitations for a specified observer. This enables color reproduction without reproducing the spectrum, but a fixed display can match only colors within its gamut. Matches between reflecting materials can fail when the illuminant changes.
L3.3metamerism can break when the illuminant changeswhere →⧉
A metameric match is a match between two products of reflectance and illuminant (L2.6). Swap the illuminant and you multiply by a different spectral shape, so two surfaces that looked identical under one light can diverge under another: illuminant metamerism. It is the headache behind prints that shift between the proofing booth and the living room, and fabrics that match in the shop but clash in daylight; color-critical work must always name the light a match is judged under.
L3.4color vision is multi-stage: trichromatic, then opponentwhere →⧉
Cone excitations are followed by neural comparisons and adaptation. Sum-and-difference models illustrate this processing, but their axes are not exact perceptual hue axes. Engineering luma/chroma transforms exploit related sensitivity differences without reproducing the neural pathway.
L3.5primary colors are a choice, not a fact about lightwhere →⧉
Specify the purpose when naming primaries: sources whose light is added, inks or filters that absorb light, or perceptual hue judgments. RGB and CMY describe common engineering choices, not unique sets required by nature. The chosen spectra and mixing process determine what can be reproduced.
L3.6Human color is non-orthogonal *and* non-negativewhere →⧉
Cone sensitivity curves overlap, and physical spectra and emitted primary powers are nonnegative. Cone responses are measurements, not spectrum coefficients. Matching a response triple with three primary lights requires solving for their powers; a negative solution identifies a color outside those primaries' additive gamut.
3.3 Adaptation and constancy
L3.7the eye's dynamic range is enormous over time and modest at any instantwhere →⧉
Distinguish sensitivity across adaptation states from the range visible in one scene under specified conditions. Local adaptation, glare, spatial structure, and the task all affect that range. A single stop count is insufficient to compare an eye, camera, and display.
L3.8vision is largely invariant to multiplicationwhere →⧉
Shared multiplicative changes preserve ratios and often leave surface appearance approximately stable. Brightness, visibility, and color can still change. Gain-control models are useful within their assumptions; they do not make vision insensitive to illumination.
L3.9ratios and log differences cancel a common multiplierwhere →⧉
Dividing a value by a similarly scaled reference cancels a common multiplier. Taking logarithms turns the ratio into a difference. This explains why log coordinates are useful for relative changes, without claiming that vision uses a global logarithm or that HDR storage requires one.
L3.10contrast depends on a reference levelwhere →⧉
Detectable increments often scale approximately with the background in the photopic range. State which contrast definition, stimulus, and viewing conditions a measurement uses. Absolute pixel error and perceived difference answer different questions; neither replaces the other.
L3.11surface appearance can remain stable as illumination changeswhere →⧉
Surface-lightness and color judgments can remain relatively stable as illumination changes. This constancy is partial and uses several cues. It differs from recovering the complete reflectance spectrum, and it does not imply that brightness or the appearance of lighting stays fixed.
L3.12the eye discounts the color of the illuminantwhere →⧉
Chromatic adaptation and scene interpretation help stabilize surface color across illuminants. Per-channel cone gains are a useful approximation. Camera white balance makes a related rendering choice for the output conditions, with accuracy limited by the measurements and estimated illuminant.
L3.13capture and viewing conditions both affect color appearancewhere →⧉
A photograph is viewed under conditions different from those at capture. Its appearance depends on the display or print, the surround, adaptation, and cues within the picture. White balance and rendering should target those conditions and the photographer's intent; viewers do not automatically undo the captured illuminant.
3.4 Spatial and temporal vision
L3.14contrast sensitivity is frequency-tuned, and it lives per degreewhere →⧉
The eye is not equally sensitive to every spatial frequency. Luminance sensitivity is band-pass, peaking at a few cycles per degree and falling away on both sides, and the curve moves with light level, field size and contrast. It is defined per degree of visual angle, not per pixel, so viewing distance and display size decide which frequencies a reader can resolve at all. Quote a peak from a textbook and you have described one experiment's conditions; measure it in the conditions you ship in.
L3.15chrominance resolves less than luminancewhere →⧉
The chromatic contrast-sensitivity functions are low-pass where the luminance one is band-pass, and they cut off at a far lower frequency, so fine color variation is much less visible than fine luminance variation. Chroma subsampling is the engineering that spends that gap. It describes typical content under typical viewing, not a law: colored lettering, isoluminant boundaries and saturated fine patterns can make the loss plain. Judge it on the content you actually have, in the viewing conditions you actually ship.
L3.16chromatic acuity depends on sampling and processingwhere →⧉
Spatial sensitivity depends on optics, receptor sampling, and neural processing. Chromatic detail often supports less resolution than luminance detail, but this cannot be deduced from opponent subtraction alone. Use measured sensitivity and inspect the intended images.
L3.17vision has a finite temporal bandwidthwhere →⧉
Temporal visibility depends on stimulus and viewing conditions. Flicker fusion does not determine a universal frame rate for smooth motion. Distinguish exposure duration, distinct image samples, display refresh, and light modulation when designing or evaluating video.
L3.18low-level perception is quantitative: it has analytic models you can evaluatewhere →⧉
Psychophysical models make quantitative predictions for specified tasks and conditions. Use the documented stimulus units, calibration, and validity range. A predicted detection threshold is not a guarantee that an arbitrary image artifact will be invisible.
3.5 The visual system attends little and makes things up
L3.19vision is foveal and serial: the eye scanswhere →⧉
Eye movements bring selected regions onto the high-acuity fovea. Peripheral signals also support scene understanding and guide the next fixation. Gaze depends on the task as well as image features; a fixation map is not a complete map of attention.
L3.20seeing is a construction, not a recordingwhere →⧉
Perception combines incomplete measurements with context, attention, and inference. Change blindness, inattentional blindness, and crowding expose limits on what we notice and retain. They do not justify treating unfixated image regions as unobserved or irrelevant.
Part 4 — Color spaces and color technology
Color Spaces And Color Technology
L4.1pixel numbers are meaningless on their ownwhere →⧉
An RGB triple needs a color-space definition and a transfer function. Processing also needs its scale and image state: camera measurements, scene-referred values, and display-referred values support different operations. Preserve this information alongside the array. A floating-point data type alone does not tell you whether the values are linear or what their primaries are.
4.1 Linear color reproduction and CIE XYZ
L4.2color analysis and synthesis require different spectral distributionswhere →⧉
Sensitivity curves measure a spectrum; emission primaries construct one. Reusing the sensitivity curves as primaries and the measured responses as weights generally gives the wrong color. Calibrate how each primary stimulates the observer, then solve for its weight.
L4.3linear tristimulus coordinates are a matrix apartwhere →⧉
XYZ, a compatible LMS system, and linear RGB are different coordinates for a fixed observer’s three-dimensional color matches. Nonlinear RGB encodings and perceptual spaces add specified nonlinear transforms. A real camera’s raw measurements usually require an approximate color correction; an appearance model also needs viewing conditions.
4.3 Encoding: linear, gamma or log
L4.4match the display interface’s expected encodingwhere →⧉
A typical sRGB image file contains encoded values. Decode them for linear-light calculations, then supply the representation expected by the output interface. A viewer expecting sRGB needs sRGB-encoded output; a linear rendering surface may encode automatically. Encoding twice is as wrong as omitting it. RAW, log, HDR, and floating-point files require their own declared interpretation.
Luma is a weighted sum of specified nonlinear RGB components. Luminance is a photometric quantity obtained from linear values with the appropriate primaries and scale. Weighting and nonlinear encoding generally do not commute. Neither constant luma nor constant luminance guarantees equal perceived brightness across colors.
L4.6additive vs multiplicative → choice of encodingwhere →⧉
Use linear light for an operation that adds light. Use ratios or log differences when the model concerns multiplicative changes. Choose a perceptual encoding for storage precision or an edit defined in that representation. Log sharpening can be a legitimate contrast edit; it simply implements a different operator from sharpening linear light.
L4.7averaging colors belongs in linear lightwhere →⧉
To average light, decode, combine linear values, and encode for the output interface. For nonnegative weights summing to one and a convex decoding curve, averaging encoded values produces no more light than averaging the decoded inputs. Kernels with negative weights need separate analysis. For alpha compositing, linearize color and use the correct premultiplied-alpha convention; alpha itself is not gamma-encoded color.
4.4 RGB color spaces
L4.8there is no single RGB: gamut versus bit budgetwhere →⧉
Choose a working gamut large enough for the content and operations, with sufficient numeric precision. Choose a delivery space the receiving system can interpret. Wider primaries alone do not improve an image, and converting an sRGB image to a wider space does not recover colors previously clipped.
4.5 Advanced color: opponents, perceptually-uniform, and preferred
L4.9distinguish chroma magnitude from relative saturationwhere →⧉
A color-difference magnitude depends on signal scale. In a linear opponent representation, scaling luminance and both centered chroma components together preserves the underlying chromaticity. Ratios describe that algebra, but a ratio such as chroma divided by luminance is not a universal perceptual saturation formula. State what the edit preserves.
L4.10equal steps in a color space are not equal perceived differenceswhere →⧉
$XYZ$, $xy$, RGB, and HSV are built to measure and store color. Judging how different two colors look takes a space deliberately warped for that job, such as CIELAB () with its $\Delta E$. Mixing the two up is why gamut areas on the $xy$ diagram mislead, why a gradient that is even in RGB banks unevenly to the eye, and why an error computed in the wrong space rewards the wrong thing. Ask of any color number: is this for measuring, or for judging?
4.6 Sensing color: multiplexing strategies
L4.11one value per photosite; color must be multiplexedwhere →⧉
Strip away the color filter and a sensor is monochrome at the pixel: each photosite reports a single number, a signal proportional to detected photoelectrons within its operating range, not a color. The three (or more) values that color needs must therefore be multiplexed along some other axis: over space (a color-filter array such as Bayer, paying with resolution and a demosaicking step), over time (sequential color filters, paying with motion robustness: the old color-wheel and many scientific cameras), across multiple sensors (a beam-splitter prism feeding three chips, the 3-CCD video camera, paying with bulk and cost), or over depth (wavelength-dependent absorption in stacked photodiodes, the Foveon, paying with noise). The choice determines which measurements are simultaneous and co-located. The same constraint shapes the eye, whose three cone types sample color over space at the retina (L3.1, L3.6).
4.7 Synthesizing color
L4.12additive primary chromaticities form a convex hullwhere →⧉
Nonnegative mixtures of fixed additive primaries lie within their chromaticity hull. Three primaries give a triangle. Maximum drive and black level also constrain the full three-dimensional gamut. Subtractive printing is not described by the same triangle rule: ink interactions and the paper must be characterized.
A stop is a factor of two. Time steps by two and f-number by $\sqrt{2}$. ISO and a software exposure slider can change rendered brightness by a stop without collecting another photon.
5.6 Preparing photographs for viewing
L5.2the room sets the dynamic range as much as the panel doeswhere →⧉
Reflected ambient light raises the visible black level. Prepare and judge the image under its intended viewing conditions; a panel specification alone does not describe the contrast the viewer will see.
Part 6 — Linear methods and Fourier analysis
6.1 Introduction to forward and inverse linear methods
L6.1a linear operator is a weighted sum, so it is a matrixwhere →⧉
An operation $F$ is linear when $F(a x+b x')=aF(x)+bF(x')$. For finite sampled signals, each output is a weighted sum of inputs with weights independent of the data. Stacking those weights into rows gives $y=Ax$. The matrix describes the operation; code can apply its weights without allocating the matrix.
L6.2undoing a local operation is not itself localwhere →⧉
The forward blur reads three neighboring samples, but its inverse uses measurements across the signal. Local operators can have nonlocal inverses, as deblurring and gradient integration illustrate. Special operators, such as pointwise scaling and shifts, are exceptions. Efficient linear solvers describes how to solve large systems without storing their inverses.
L6.3the forward model is easy; the inverse is the hard partwhere →⧉
For a known linear model, predicting the measurement is $y=Ax$. Recovering $x$ requires checking uniqueness, noise sensitivity, and the model assumptions. If the measurement loses information, a solver alone cannot restore it; recovery needs additional data or assumptions about the scene.
6.2 Convolution
L6.4a convolution is a shift-invariant weighted sumwhere →⧉
A convolution uses fixed weights that depend on displacement, not on image values or absolute position. On an infinite or periodic grid it is linear and commutes with shifts. The kernel can be small or span the image. Its support determines the work of direct filtering; finite boundaries determine where the shift-invariance model applies.
L6.5convolution is a weighted sum of shifted imageswhere →⧉
Convolution can be computed per output pixel or per kernel tap. In the first order we accumulate a neighborhood; in the second we accumulate shifted copies of the image. The result is the same. Replacing shifts with known warps gives other linear operators, but they need not share convolution’s commutativity or Fourier diagonalization.
Finite differences are convolutions with signed kernels under a declared indexing convention. In the continuous setting, $\partial_x(gI)=(\partial_x g)I$: convolving with a derivative-of-Gaussian differentiates a smoothed image. Gradient components are linear; gradient magnitude, thresholding, and a complete edge detector generally are not.
L6.7most imaging blur is not a fully shift-invariant convolutionwhere →⧉
A convolution is an approximation when blur changes with depth, motion, field position, or wavelength. A small patch may support one kernel, but occlusion boundaries need particular care. Fixing geometry can preserve linearity in intensity even when shift-invariance is lost.
Express a linear operator in a basis where it becomes diagonal. A coupled, hard-to-invert operation in one representation can be a list of independent scalar multiplications in another, and scalars are trivial to understand, compose, and invert. Fourier diagonalizes convolution, turning a blur into per-frequency multiplication. This is not a one-off trick: it is the single most reusable move in applied linear algebra, and it returns for differential equations (the heat equation, below), for principal components, and for nearly every inverse problem in this book. The idea is already visible in the smallest possible case, a $2\times2$ matrix (a figure).
L6.9phase aligns the waves that form structurewhere →⧉
Phase specifies the alignment of the waves; magnitude specifies their strength. Phase swaps between typical photographs often preserve recognizable structure from the phase donor, but accurate reconstruction requires both.
L6.10natural-image spectra often fall approximately as $1/f$ in amplitudewhere →⧉
Natural photographs often have approximately $1/f$ average Fourier amplitude over a range of scales. Distinguish amplitude from power, and average power per coefficient from total energy in a frequency band. The image mean can dominate total energy without explaining its visible detail.
L6.11the boundary model determines which convolution the FFT computeswhere →⧉
The DFT diagonalizes circular convolution exactly. With sufficient zero padding it also computes finite linear convolution exactly. Each basis wave is global, while the complete spectrum retains all spatial information. Choose the boundary extension to match the image operation.
L6.12convolution is per-frequency multiplication, so inverting it divides, and amplifies noise where the response is smallwhere →⧉
In the Fourier basis a convolution is diagonal: it multiplies each frequency by one number (the kernel's complex response, the OTF), so undoing it is a per-frequency division. The whole stability question becomes local to each frequency: how near zero is that number? Blur is low-pass, so its response is tiny at exactly the high frequencies it destroyed; the lower a frequency's coefficient, the more its noise is amplified on inversion. That is why naive deblurring fails worst precisely where the blur was strongest, and why a Wiener / regularized filter stops dividing once the signal drops below the noise floor.
6.4 Sampling, downsampling, and aliasing
L6.13Nyquist–Shannon: sample above twice the highest frequency, or it aliaseswhere →⧉
Sample above twice the highest retained frequency. If the signal is too detailed for the grid, low-pass it before sampling. This applies to space and time. Real prefilters have finite transition bands, so allow a margin below Nyquist and check residual aliasing.
L6.14sampling replicates the spectrum; aliasing is the replicas overlappingwhere →⧉
Sampling on a grid is multiplication by a comb (a train of deltas), and a comb's Fourier transform is another comb, so sampling replicates the signal's spectrum at every multiple of the sampling rate. If the spectrum is band-limited below Nyquist the copies sit side by side and the original is recoverable; if it is not, the copies overlap, and the folded-over tails land on top of the genuine low frequencies as aliasing: energy appearing at completely wrong frequencies (moiré, the backward wagon-wheel). The cure is to band-limit before sampling: low-pass away everything above Nyquist first, then sample.
L6.15the ideal sampling filter is a sinc, which does not exist in practicewhere →⧉
Ideal band limiting and ideal interpolation use sinc kernels, scaled for their respective grids. Their infinite support requires approximation in finite computations. A finite filter trades computation and frequency selectivity; inspect both aliasing and edge overshoot.
L6.16a resampling / anti-aliasing filter trades the ideal sinc against compact supportwhere →⧉
Fourier analysis says the perfect reconstruction and anti-aliasing filter is the sinc, a perfect box in frequency. But the sinc is infinitely wide and rings when truncated, so no real filter can be it. Every practical kernel (bilinear, bicubic, Lanczos) is a deliberate compromise between approximating that idealized sinc (what Fourier wants) and staying compactly supported (what computation, and ringing, demand): reach wider and you approach sinc; truncate harder and it gets cheaper and less ringy but further from ideal.
L6.17choosing a resampling filter trades passband droop against stopband leakwhere →⧉
Read off the frequency response, the same compromise is a tug-of-war between two errors. Passband droop: a too-gentle filter attenuates frequencies below Nyquist that the grid could have represented, so the image goes soft. Stopband leak: a too-aggressive filter lets frequencies above Nyquist through, which alias into jaggies and moiré. A good kernel sits at the sweet spot (flat passband up to Nyquist, sharp rejection beyond) and box, tent, bicubic and Lanczos are just different points on this droop-versus-leak curve.
L6.18bicubic (and friends) are convolution with a fixed kernel, not polynomial regressionwhere →⧉
The cubic-convolution filters used here evaluate a fixed piecewise-cubic kernel at fractional offsets and sum weighted samples. They do not solve a regression for every output pixel. Other methods also called bicubic interpolation use polynomial or spline constructions; the name alone does not identify the algorithm or its downsampling behavior.
6.5 Resampling and upsampling
L6.19resample by gathering, rarely by scatteringwhere →⧉
Loop over output pixels and pull back through $f^{-1}$ to find where each one reads from. Gathering fills every output exactly once; scattering pushes input pixels forward and leaves holes whose shape depends on the transform, a lattice of gaps under a scale and tilted slivers under a rotation. The same reflex governs warping, morphing, texture mapping, and flow-based reprojection later. Scattering earns its place only when the forward map is the thing you actually have, as in splatting a point cloud, and then it needs an explicit accumulation and normalization pass.
L6.20every resampling pass multiplies the spectrum again, so resample oncewhere →⧉
Repeated resampling accumulates blur, ringing, and sampling error. For repeated fixed convolution the frequency response is exactly $H^N$; for rotations it is only an illustration because the grid and effective filter change at each pass. Compose geometric transforms and resample the original once.
The Fourier chapter's lesson was diagonalize when you can: work in the basis where your operator is simple. This chapter adds the practical sequel: the perfect basis is rarely the useful one. Fourier is perfectly localized in frequency and not at all in space; the primal grid is the reverse. The uncertainty principle (compact in space ⇔ spread in frequency, from the Fourier chapter) says you cannot have both perfectly. A pyramid surrenders a little of each (it chops frequency into a few coarse bands and keeps each band laid out spatially), and that modest sacrifice is exactly what makes per-scale, per-region editing possible. Reach for a representation that is local in both, not for the one that is perfect in one.
6.7 Image formation as a linear system
L6.22what makes recovery hard: a lossy operator and additive noisewhere →⧉
Recovery depends on which variations the operator measures, their signal-to-noise ratios, and how accurately the operator is known. A null space leaves ambiguity; small gains make inversion sensitive to noise. An unknown or miscalibrated operator adds a separate estimation problem.
Minimize a data mismatch and, when needed, a regularization penalty. Unregularized least squares agrees with exact inversion for a square invertible matrix. Regularization deliberately changes that estimate, trading data fit against a preference for particular solutions. Existence and uniqueness depend on the chosen objective.
L6.24a camera is a giant linear operator $\mathbf{y}=A\mathbf{x}+\mathbf{n}$where →⧉
Strip away the glass and an imaging system is one linear map: it takes the scene $\mathbf{x}$ (written either as a flat billboard at a fixed depth or, more generally, as ray space, the light field) and returns the image $\mathbf{y}=A\mathbf{x}+\mathbf{n}$, a linear projection of the scene plus sensor noise. The matrix $A$ is the camera: pinhole, lens, stereo, coded aperture, the motion-invariant sweep, they differ only in their rows. And because $A$ is a design rather than a fact of nature, building an optical system is choosing $A$: the whole of computational optics is engineering the operator, in space, in time, in ray space, together with the reconstruction that inverts it, to pull better images and extra information out of the light. Everything after this chapter is organized by one question about that operator: how invertible is $A$?
Apply the forward operator, its true adjoint, and the regularizer without forming a dense matrix. Use CG only for a symmetric positive-definite system, and report a residual-based convergence check. Exploit Fourier or multigrid structure when the model supports it.
Part 7 — Probabilistic Approaches and Machine learning
7.1 Imaging as Bayesian inference with probabilistic priors
L7.1natural images are a tiny sliver of pixel spacewhere →⧉
Natural images occupy only a tiny fraction of the space of all arrays of RGB values.
L7.2model the manifold, rule out the impossiblewhere →⧉
If we can computationally model the manifold of natural images, we can use it to exclude unlikely solutions to inverse problems.
L7.3every restoration method embeds a prior, named or notwhere →⧉
A bilateral filter, an edge-directed demosaicker, and an end-to-end network each bet on what natural images look like. One states the bet as a penalty, one hides it in the machinery, one absorbs it from data, and none of them can avoid making it, because the measurement alone does not determine the answer. The only choice is whether the bet is deliberate and inspectable. Faced with any method in the chapters ahead, locate its prior first: it tells you which images it will quietly fail on.
L7.4a regularizer is the negative log of a priorwhere →⧉
With Gaussian measurement noise, a negative log prior supplies the regularizer in a least-squares objective. A Gaussian prior gives a quadratic penalty; sparse priors give other penalties. Check whether the proposed prior is normalizable and what image variations it suppresses.
L7.5factorization is inverting a product, and it is fundamentally ill-posedwhere →⧉
Factorization problems seek to decompose a number, or an image, into a product of two unknown pieces (scalars, or whole matrices) given only the product. This is fundamentally ill-posed: many factor pairs multiply to the same result, and unconstrained formulations often admit the degenerate "put everything in one factor, leave the other the identity" split. Additional constraints, calibration, or priors breaks the tie. In the fully general case it is our linear problem $\mathbf{y}=A\mathbf{x}+\mathbf{n}$, except we must now solve for the operator $A$ as well as the unknown $\mathbf{x}$. Examples include white balance, intrinsic images, matting, and blind deblurring.
7.2 Machine learning
L7.6a learned operator swaps a hand-designed prior for one learned from datawhere →⧉
A learned method can fit a prior or a complete restoration operator from examples. The examples and loss determine what it learns to preserve or infer. An explicit measurement constraint is retained only if the method includes one; a plausible learned output need not satisfy it.
7.4 Generative AI and diffusion
L7.7a generative model is a sampleable prior over imageswhere →⧉
A generative model can draw images from a distribution. Conditioning on a measurement can produce several plausible reconstructions of an ambiguous input. Sampling does not establish that their details were present in the photographed scene.
When several scenes explain the measurement, an image prior helps select or weight them L7.8. A generated reconstruction remains one explanation among others.
Part 8 — Basic image processing and ISP
8.1 Image representation
L8.1Dither: trade structured error for unstructured noisewhere →⧉
Dither replaces visible quantization structure with noise. Independent stochastic rounding produces approximately white noise; error diffusion shapes the error spatially and can leave patterns. Dither preserves an average under stated assumptions, not every pixel or every local region.
L8.2loop in memory order (innermost index varies fastestwhere →⧉
Prefer traversal that reuses nearby stored values. For scalar loops over packed HWC, use y, x, c order; for packed CHW, use c, y, x. Actual strides, tiling, and vectorized kernels can change the best schedule. Measure before changing the representation.
8.3 Tone point operations
L8.3exposure scales linear light (a multiply) or shifts log values (an addwhere →⧉
Exposure scaling multiplies linear-light values by $k=2^s$ or adds $s$ in base-2 log units. For a pure gamma encoding $V=L^{1/\gamma}$ it multiplies V by $k^{1/\gamma}$. For an arbitrary transfer function, decode, scale, and encode. Output clipping is a separate operation.
L8.4contrast is a pivot: choose what stays putwhere →⧉
A contrast change rotates values about a chosen pivot: $L_\text{out} = p + g\,(L_\text{in}-p)$. The pivot $p$ is exactly what is left unchanged, and it also decides whether the picture gets brighter or darker overall: a high pivot pulls most of the image down (darkening), a low pivot pushes it up (brightening). Unlike exposure, contrast is not radiometrically meaningful, it is a stylistic stretch, so the encoding matters less (no single space is "correct"); what matters is a sensible pivot, and the sensible pivot does depend on the space: roughly 18% mid-gray in linear light, the numeric midpoint in gamma or log. Pick the pivot deliberately: it is the whole character of the operation.
8.4 Beauty curves: the camera "look"
L8.5radiometrically correct is not the same as pleasingwhere →⧉
Calibration and appearance are separate decisions. Preserve the calibrated source and specify the rendering transform. A pleasing result need not preserve scene radiance ratios, but later code must not mistake those rendered values for the original measurement.
Preserve original samples and store the operations and parameters separately. This makes adjustments revisable without repeatedly quantizing intermediate images. Also preserve the processing version and dependencies; a recipe is reproducible only while its ingredients and interpretation remain available.
Part 9 — Evaluation
9.2 The expert's eye as evaluation
L9.1you are an expert on the artifact, not on preferencewhere →⧉
Use expertise to identify and explain defects. Claims about how often viewers notice them, how much they matter, or which result people prefer require evidence from those viewers.
L9.2average performance hides the failure that shipswhere →⧉
Report the distribution and inspect its tail. A better average can coexist with a serious regression; record the frequency, severity, and conditions of the failures that matter to the intended use.
9.3 Image metrics
L9.3$L_2$ is everywhere for two reasons, and neither is “it matches perception”where →⧉
Squared error is smooth and mathematically convenient, and strongly penalizes large residuals. For a linear model, unconstrained least squares leads to a linear system. A neural network trained with the same loss remains a nonlinear optimization problem. Under squared error, the optimal prediction of an uncertain target is its conditional mean, which explains the averaging discussed in Slopendium/Denoising.
L9.4when a metric becomes a loss, check its blind spotswhere →⧉
Optimizing a score can improve what it measures while exploiting what it ignores. The score does not automatically become useless, and evaluating a training loss on held-out data still measures generalization of that objective. But it is weak independent evidence for a broader quality claim. Reserve evaluation data, inspect outputs, and use complementary measures or human judgments that match the claim. Repeated tuning against the “held-out” measure makes it another target.
9.4 Towards meaningful user studies
L9.5Run a study only if its outcome would change a decisionwhere →⧉
Name the question and the decision it informs. A study can guide a design, test a scientific claim, or reveal an unmet need. A required table of numbers is not a research question.
Use the color / intensity difference between two pixels as how much they belong together, an affinity, then average (or connect) pixels in proportion to it. Pixels with high affinity are treated as measurements of the same underlying thing and get averaged together; pixels with low affinity, across an edge, are kept apart. The bilateral filter's range weight $g_r$ is the first full instance, but the same affinity drives the bilateral grid, the guided filter, non-local means (NL-means), colorization, and matting / segmentation that follow. Edge-preserving is affinity. (This is L10.1, registered in Denoising where the bilateral first appeared for noise; this is its full edge-preserving treatment.)
L10.2discretize the range to accelerate a non-linear filterwhere →⧉
The bilateral is awkward because it is non-linear: its weights depend on the pixel values, so they differ at every output pixel and the filter is not a convolution. The bilateral grid's deeper trick, beyond "add a dimension", is to discretize the intensity (range) axis into a small set of levels, and then do cheap, ordinary linear work at each level. Quantize $I$ into grid layers, run a plain blur within and across layers (distant layers interact less, with accuracy controlled by grid resolution and the blur), and interpolate back to each pixel's exact value. The per-pixel nonlinearity becomes "a few linear passes plus a lookup." This is a general recipe, not a bilateral-only hack: the fast local Laplacian filter (Local Laplacian filters, Aubry et al. 2014) accelerates its per-pixel nonlinearity the same way: evaluate a handful of discrete intensity levels of a slow operator and interpolate between them. Whenever a filter's cost comes from a continuous, value-dependent response, discretizing that response axis turns it into reusable, parallel, convolution-shaped work. (Registered in Big Lessons as L10.2; recurs in Local Laplacian filters and, as an algorithm-level speedup, in Algorithmic speedups.)
L10.3an extra dimension lets you downsample space without losing edges, and makes big kernels cheapwhere →⧉
Lifting the image into the higher-dimensional grid $(x, y, I)$ is what makes the coarse grid safe. Because pixels on opposite sides of an edge sit at different intensity heights, they are already separated along the range axis, so you can downsample the spatial axes aggressively while retaining range information that ordinary spatial downsampling would lose; coarse sampling can still distort boundaries. Two gains follow: high-resolution edges survive despite a coarse grid; and a large spatial kernel becomes a small blur on the coarse grid, so larger spatial widths may reduce grid work, although full-resolution splatting and slicing still cost at least one pass over the image. Paying one extra dimension for cheap, edge-aware, large-scale smoothing is a trade that recurs: NL-means' patch space, the permutohedral lattice, and the learned bilateral grid (HDRnet) below all live on it.
The eye cares about local contrast, not absolute values, so edit the contrast directly, in the multiscale decomposition. Where gradient-domain methods (Poisson image editing) manipulate the gradient field and re-integrate, local Laplacian filtering manipulates the Laplacian-pyramid coefficients (the local contrast at each scale) with a per-pixel point curve, then collapses the pyramid. Both discard or reshape contrast while letting the absolute level fall out, and both are kin to the bilateral base/detail split; the difference is that working band by band with an identity tail for large contrasts removes the mechanism that makes a single base layer halo: no base value is committed once for the whole edge. In practice the halos shrink to the point of being hard to find at moderate settings, though extreme parameters can still produce small artifacts near strong edges. (L10.4, registered in Poisson image editing; this is its multiscale-decomposition appearance.)
10.9 Seam optimization
L10.5image edits as discrete optimization on a graphwhere →⧉
Many image edits are discrete optimization on a graph or grid, whether dynamic programming (a least-cost path), min-cut / max-flow (a globally optimal boundary), or a spectral relaxation (normalized cuts), and the energy (affinity) you choose is the whole design. The optimizer is generic and off the shelf; what you penalize (gradient magnitude along a seam, label disagreement across an edge, the cost of severing two regions) is the entire design. Read this as the L10.1 affinity principle from bilateral chapter turned inside out. There, the affinity between two pixels said "average these together." Here, the very same quantity, sitting on a graph edge, says "do not cut here." High affinity (similar neighbors) marks an expensive place to cut; low affinity (a genuine edge) marks a cheap one. Design that cost well and a generic min-cut often returns a boundary close to the one a person would have drawn; when it does not, the fix is usually in the energy, not in the optimizer. (Registered as new lesson L10.5 · image edits as discrete optimization on a graph (the energy is the design), first appearance here. Extends L10.1. Also relevant downstream: graph-cut photomontage composed with a Poisson blend; compositing / segmentation / matting; texture synthesis; stereo and Markov-random-field labeling.)
Part 11 — Warping and morphing
11.5 Shape-preserving warping
L11.1*to preserve shape, penalize the non-rigid part of the local motion*where →⧉
A free-form warp that only interpolates handle positions will happily shear, stretch, and collapse a shape, because position-interpolation has no opinion about how a region deforms, only about where its anchors land. To make a deformation look physical you add a regularizer that measures, at every point, how far the local transformation is from a rigid motion (a rotation + translation) and minimizes it. Rotation is left free (a limb may swing any amount); only the stretch/shear part is charged. The recurring move (fit the constraints, but regularize toward a prior about the world) is the same one behind Poisson editing (regularize toward a gradient field), content-aware retargeting (regularize toward leaving salient regions rigid), and morphable models (regularize toward a learned shape basis). Here the prior is the most generic physical one there is: things rotate freely but resist stretching.
The same $2\times2$ matrix $M=\sum_W \nabla I\,\nabla I^{\top}$ answers three questions across this part, and the unity is the point. Is this a good point to detect? Yes, where $M$ has two large eigenvalues, a corner: that is Harris / Shi–Tomasi, here. Is this patch's motion solve well-conditioned? The very same matrix, now required invertible: that is KLT (Feature tracking). What is the local flow constraint? $M$ is the normal-equations matrix $A^\top A$: that is Lucas–Kanade (Optical flow). Detect with it, track with it, solve flow with it: one matrix, read three ways. Corners are special for a deep reason: the same both-directions-vary condition that localises a point is also what makes its motion recoverable. So "good to detect" and "good to track" are not two coincidentally-aligned criteria; they are one statement about $M$.
Recording several captures lets us choose or combine their measurements afterward. A filter wheel is the plain example: shoot the scene through a red, a green and a blue filter (or through a polarizer at several angles) and the choice of how to combine the bands can be made later, at the desk. Exposure brackets, focal stacks, and panoramas extend this idea to different capture settings.
L13.2multiple-exposure imaging breaks the limits of a single capturewhere →⧉
Several compatible measurements can reduce uncertainty or cover information missing from one frame. This is a different benefit from the previous lesson. Deferring a choice picks among things one exposure could have recorded; combining exposures can exceed what any single exposure could record, such as a noise level below one frame's, a dynamic range wider than the sensor's, or a depth of field deeper than the aperture allows. The reconstruction must account for differences in exposure, focus, viewpoint, or scene motion.
L13.3splitting light is free only in bright lightwhere →⧉
With a fixed total count and negligible read noise, splitting a measurement into independent exposures has little SNR cost if the photons can be recombined. Each additional read adds variance. Lost photons, motion, and new unknowns such as separate depth or color channels add costs beyond this simple accounting.
13.13 The limits of denoising
L13.4the average is safe, and the average is dullwhere →⧉
Squared-error minimization returns the posterior mean. When plausible answers disagree, their average can blur edges or desaturate uncertain colors. A perceptual or generative objective can produce a sharper alternative, but sharpness does not establish fidelity. Evaluate both properties.
Edge-directed demosaicking is a first, very concrete instance of a theme that returns throughout the book: adapt the operation to the local structure of the image rather than applying the same fixed filter everywhere. Here the structure is an edge and the adaptation is choosing the interpolation direction; later, in edge-preserving filtering and denoising, the same instinct (use the similarity between pixels to decide how much they should influence each other) becomes the bilateral filter and the idea of affinity. Whenever a fixed linear filter blurs across something it should have respected, the cure is to look at the data first.
L15.2luminance carries detail, chrominance is smoothwhere →⧉
Channel differences are often smoother than luminance-like structure, which makes guided reconstruction useful. Preserve that distinction as an image model rather than a law: fine color texture and saturated boundaries can break it.
A global curve can reveal recorded highlights while allocating useful contrast to the rest of the image. Check its effect on local contrast and color. It is inexpensive and predictable, but composition order still matters: exposure followed by a curve differs from a curve followed by exposure.
L18.2tone-map in the log domain when the operation acts on ratioswhere →⧉
Log luminance is useful when the operation should act on intensity ratios, as in additive base/detail decomposition or gradient attenuation. Other operators, including Reinhard's $L/(1+L)$, can be formulated directly in linear luminance. Match the representation to the operation; do not take logarithms of nonpositive values.
18.11 Evaluating tone mapping
L18.3state what a comparison of defaults measureswhere →⧉
Defaults are appropriate for comparing out-of-the-box behavior. For claims about algorithmic capability, specify a fair tuning budget and evaluate the frozen configurations on separate images. Report the settings and test whether the conclusion changes over a reasonable parameter range.
Part 21 — Inpainting
21.4 Generative inpainting
L21.1a stronger prior hallucinates more convincingly, not lesswhere →⧉
Weak priors fail loudly, blurring or smearing, and you discard the result without thinking. Strong generative priors fail quietly, returning something sharp, coherent and wrong. Improving the model improves the plausibility of its inventions at the same rate as the quality of its reconstructions, so the burden of distinguishing the two moves from the algorithm onto the person using it, and onto the provenance record that travels with the file.
Part 29 — Making computational photography code fast
29.2 Modern CPUs: memory hierarchy, parallelism, and what it takes to go fast
L29.1moving data can cost more than arithmeticwhere →⧉
For a kernel limited by data movement, removing intermediate traffic can help more than removing arithmetic. First establish the bottleneck: expensive functions, long dependency chains, and dense numerical kernels can still be limited by computation.
L29.2tiling: fit the working set in cache, reuse every bytewhere →⧉
Choose tiles large enough to amortize overhead and small enough to retain useful data. Include halo storage and concurrent workers. Tiling helps when it improves reuse; it can add work without benefit to a simple streaming operation.
For fixed work, an idealized speedup on $p$ workers is $1/[s+(1-s)/p]$, where $s$ is the serial fraction of the one-worker time . A 5% serial fraction caps that speedup at 20. Gustafson considers growing the workload with the worker count . These are different assumptions; neither eliminates communication, bandwidth limits, or parallel overhead.
L29.4separate the algorithm from its execution schedulewhere →⧉
The algorithm specifies the result; the schedule controls loop organization, parallelism, storage, and recomputation. Changing the schedule can change both traffic and executed arithmetic, especially at tile halos. Halide makes these choices explicit so you can compare them without obscuring the algorithm.
29.6 Hardware backends: GPU, NPU, DSP
L29.5specialization helps when the workload fitswhere →⧉
Choose the unit that executes the required operations efficiently within the whole pipeline. Include movement and conversion costs. Flexibility and efficiency trade off, but CPU, GPU, DSP, NPU, and ISP do not have a universal ordering of performance per watt.
Part 33 — Computational optics and coded imaging
33.5 Code in time (phase, amplitude)
L33.1the same operations recur on every axis of the plenoptic functionwhere →⧉
The usually-forgotten thesis of Adelson & Bergen's plenoptic paper (→ Image measurements as integrals) is that low-level imaging does one kind of thing, repeated across every dimension: space, time, aperture, wavelength. This chapter is that thesis turned into engineering. Coding the exposure in time (the flutter shutter, the parabolic sweep) is the same move as coding the aperture in space (Coded aperture), as the phase plate of Wavefront coding, as focal-sweep extended depth of field, as light-field refocusing across the aperture, as chroma tricks across wavelength. Sample-or-integrate, code-or-leave-clear, make-the-blur-invertible-or-invariant: pick a slice of the plenoptic function and the toolkit is the same. So when a new imaging problem arrives, ask which axis it lives on and which of these moves applies: the answer is usually already in the book, sitting on a different axis. (Light field across the aperture is what high-speed video is across time; a coded aperture is what a flutter shutter is in time.)
L34.1dynamic range has a *second* floor, one photonwhere →⧉
L2.46 set the bottom of a single exposure at the read-noise floor, the dimmest signal that survives the amplifier and ADC. But that floor is an engineering limit, not a physical one. Drive a photodiode past breakdown (a Geiger-mode avalanche) and a single photo-electron makes a macroscopic, digitally-countable click: the floor drops to one photon, read noise effectively zero, and the only noise left is the shot noise of the light itself (L2.38). So dynamic range is bounded at the top by full-well capacity and at the bottom, ultimately, by the discreteness of light, not by the chip. Single-photon sensors live at that opposite extreme: where an ordinary pixel sees only its own read noise, they count individual arrivals.
Velocity in a video is almost always inferred: track a feature or solve optical flow across two frames and divide displacement by time, producing an estimate that needs texture, good light, and an aperture-problem-free edge, and that arrives a frame late. A Doppler sensor skips the inference and measures radial velocity directly, in one instant, from the frequency shift the moving surface imprints on a coherent (laser/radar) return. The advantage of a direct measurement is familiar: it is fast (no two-frame baseline), works in low light and on textureless surfaces (it rides the active illumination, not scene contrast), and it reads the component optical flow cannot, motion along the line of sight. The blind spot is the mirror image: it is deaf to motion purely across the frame, which is exactly optical flow's strength.
34.7 Event sensors
L34.3report change, not state: the asynchronous pixelwhere →⧉
A conventional sensor is synchronous and absolute: every pixel reports its brightness, on a shared clock, every frame, including most pixels that did not change, paying full data, power, and latency to re-send a static world, and smearing whatever moved into motion blur. An event sensor (and, in its own way, single-photon) inverts this: each pixel is asynchronous and differential, staying silent until something happens and then firing a timestamped message the instant it does. The result is structural: microsecond latency, no motion blur, huge dynamic range, and data/power that scale with scene activity, not pixel count. The cost is equally structural: there is no frame and no absolute intensity, so the entire downstream toolkit, built on dense, synchronous arrays, must be rethought for a sparse, asynchronous stream.
Part 40 — Human factors
40.1 Human factors and the art of photography
L40.1Separate the subject from its surroundings, and commitwhere →⧉
The recurring move of making a good picture is separation: lift the subject out of its background so the eye knows, instantly and without effort, what to look at. You have four levers to do it: depth of field (throw the background out of focus), framing and viewpoint (find a clean background, build layers front to back), light (a rim light, a pool of brightness, contrast against shadow), and color (a warm subject on a cool ground, or a saturated subject on a desaturated one). Pick a lever, then commit. Whatever you do (off-center the subject, blur the background, tilt the horizon, push the saturation), do it boldly or not at all. The timid half-measure, the subject that is almost off-center, the background that is sort of blurred, is the amateur's tell. Most of a photograph's craft is deciding what to leave out, and then leaving it out without flinching. Every rule in this chapter is one tactic in service of this single idea.
40.6 The social and personal practice of photography
L40.2photography supports memory and communicationwhere →⧉
Immediate sharing adds uses to photography. Determine whose activity is being studied before claiming that one motive has replaced another.
Part 42 — Systems
42.5 Networking and image transport
L42.1moving pixels costs more than computing on themwhere →⧉
Moving an image, across a network or through a memory hierarchy, usually costs more than the arithmetic done on it. Compute where the pixels already are, and when the computation must happen elsewhere, send as little as possible: a smaller image, a description of the result, or instructions for producing it.