• start with a 1-D signal (a single row of pixels) and the simplest blur: a **box blur** that replaces each sample by the average of itself and its neighbors

• write it out explicitly as a small matrix–vector product $\mathbf{y}=A\mathbf{x}$: each output sample is a weighted sum of input samples, so the box blur is a matrix $A$ with a band of equal weights along the diagonal

• this is the **forward model**: given the scene $\mathbf{x}$, produce the blurred measurement $\mathbf{y}$. Keep the example tiny (a handful of samples) so $A$ can be written on the page and checked by hand

• the natural question: given only the blurred $\mathbf{y}$, can we recover the sharp $\mathbf{x}$? This is **deblurring**

• the point of the chapter: inverting the blur is **just linear algebra** — solve the linear system $\mathbf{y}=A\mathbf{x}$ for $\mathbf{x}$, i.e. invert (or pseudo-invert) the matrix $A$

• foreshadow the trouble without diving in: $A$ can be ill-conditioned or singular, so naive inversion amplifies noise; the full story waits for [[Image formation as a linear system]] and [[Efficient linear solvers]]

• a striking real-world instance of the same forward/inverse pair: the **corner camera** (Bouman et al., *Turning Corners into Cameras*, 2017), which recovers a hidden scene *around a corner* from subtle shading it casts on the floor

• the edge of a wall acts as a crude one-dimensional aperture: the **penumbra** on the ground is a blurred, angularly-integrated measurement of the light from the hidden scene — again a forward model $\mathbf{y}=A\mathbf{x}$

• recovering the hidden scene is a **linear inverse problem**, solved with the same tools introduced above; a memorable demonstration that inverting a linear forward model can *see the unseen*

• a pixel computed from its neighbors (recall the three operation types)

• blur also comes from optics: diffraction, spherical aberration → motivates removing optical artifacts (forward ref: deblurring) so important to understand if we want to prevent and remove it.

• blur as a weighted average of neighbors.

• linear shift-invariant (LSI) filtering; why linearity matters (with a non-shift-invariant counter-example)

• the convolution algorithm: loop over the output

• point spread function (PSF) / impulse response

• **debugging / hand-checkable inputs**: an **impulse** reads off the kernel (the PSF); a **constant** must pass through a normalized kernel unchanged; a **shift kernel** just translates the image; a **half-plane** (edge) shows the edge response; verify a **1-D 3×3** by hand first

• without the flip it's called correlation. USeful too but not as useful and doesn't have all the good mathematical symmetries.

• sidebar: convolution in probability (sum of random variables, Laplace first derivation of convolution, flip is obvious: if we want the sum to be a given number, then the second number has to be the sum minus the first one)

• **sidebar — debugging convolution** (recipe moved here from Developing, testing & debugging): convolve an **impulse** and the output **is your kernel**, laid out and flipped — read it off number-for-number to catch a wrong weight, an off-by-one in centering, and the **flip** (convolution vs correlation) at a glance; a **constant** comes back unchanged iff the weights sum to 1; an off-center single-1 **shift kernel** translates the image by exactly that offset (tests sign & direction); a **half-plane** shows the border behavior; do the very first test in **1-D (1×3)**, where every output is hand-computable. [fig-psf-impulse]

• **normalization**: the weights sum to 1 to preserve brightness (a blur shouldn't darken the image)

• **commutativity / associativity**: I*g = g*I and (I*g)*h = I*(g*h) — so filters can be pre-combined and reordered (the flip is what buys these symmetries)

• **linearity**: scales and superposes — the very reason Fourier will help (next chapter)

• the **identity** is the impulse δ (I*δ = I); **separability** gets its own point below

• edge padding (ties to the edge/accessor policies above)

• centering the kernel

• box filter

• Gaussian (formula, truncation at ~3σ)

• a 2-D filter is **separable** if it's an outer product of two 1-D filters (the Gaussian is): blur **rows, then columns**

• big **cost** win: a (2r+1)×(2r+1) kernel costs O(r²) per pixel; done separably, **O(2r)** — the practical reason large Gaussian blurs are fast [separability figure]

• **derivative filters**: finite differences estimate ∂I/∂x and ∂I/∂y; the **Sobel** kernel pairs a derivative (one axis) with a little smoothing (the other) for noise robustness

• the **gradient** ∇I = (Iₓ, I_y) → its **magnitude** (edge strength) and **orientation** (edge direction): the basis of edge detection (forward ref: features / Canny) [gradient / Sobel figure]

• **the median filter — the simplest neighborhood op that *isn't* a convolution** (the first step off the linear path): replace each pixel by the **median** of its window. It is a **rank / order-statistic** filter — not a weighted average at all, so it can't be written as a convolution and has no frequency response. Two consequences make it the canonical nonlinear neighborhood op: (1) an **outlier can't sway it** — one wild value (salt-and-pepper / impulse noise, a hot pixel) sits at the *end* of the sorted window, far from the median → it's **rejected**, where a mean/box-blur would smear it into a gray blob; (2) it is **edge-preserving** — across a clean step, the window's median lands on whichever side holds the majority, so the edge stays sharp instead of ramping. It does round corners and erase thin lines, and it's the same robust-statistic idea as the temporal median (deghosting) and sigma-clipping — *use the consensus, not the average*. (Full denoising treatment: [[#Denoising basics]] → *Spatial averaging and its limits*.)

• a Gaussian blur **smears edges** because it weights neighbors by spatial distance alone (a near-but-different pixel pollutes the average)

• fix: weight each neighbor by **value similarity** too → the **bilateral filter** (Tomasi & Manduchi 1998): spatial Gaussian × range Gaussian on the intensity difference [edge-preserving figure]

• per-pixel normalization (weights depend on the values); **nonlinear** and **not shift-invariant** — leaves the clean world of convolution (no commutativity, no Fourier diagonalization)

• 💡 **Big lesson:** **edge-preserving = affinity** — use the color/intensity difference as how much two pixels "belong together"; the engine of edge-aware denoising/sharpening, halo-free local tone mapping, matting, colorization

• two kinds of weighted average: fixed weights → convolution (linear, shift-invariant); image-dependent weights → edge-preserving (bilateral, nonlinear)

• two threads forward: **Fourier** (next — convolution becomes per-frequency multiplication, enables deblurring) and the **affinity** lesson (recurs all book)

Why? Allows us to use linear algebra, e.g. to solve and understand inverse problems such as deblurring.

cite Strang.

Useful both conceptually (translation to the language of linear algebra) as well as practically (use linear algebra libraries and well-established solvers)

Similar to how we store in memory: serialize all pixels/channels.

Note: one source of notation overload: images are often named x.

Operations on vector spaces and what they mean for images: addition, multiplication by scalar

dot product aka scalar product aka dot product aka inner product

Matrices without forming matrices

Once images are modeled as vector, linear operators (exposure, convolution) can be represented as matrices. But giant matrices. In a case like convolution

And just like with any vector space we can do a change of basis. See next.

Convolution is a linear operation. It can be represented as a matrix. But matrices and linear operations can be unwieldy to understand. For example, we may want to know how easy it it to invert a matrix (if we want to deblur).

Said differently, the output of a convolution for a pixel depends on other pixels and these cross-dependencies create complexity.

However, there is a special case of matrix that is easy to manage : a diagonal matrix. In a diagonal matrix, there is no cross-talk, each input just undergoes a scalar multiplication, which means the inverse is a simple scalar division. The difficulty of inversion boils down to how close the scalar is from zero.

Fortunately for us, we can do a change of basis (change of image representation) so that convolution turns into a diagonal operation. That change of basis is the Fourier transform. Warning: the only bug is it assumes that the image is cyclic. But still insightful.

And we have **already seen** that frequency is a good way to model **human vision**: the contrast sensitivity function (**CSF**) back in the perception chapter (Spatial vision) describes the eye's sensitivity as a function of *spatial frequency*. So the Fourier view is not only a computational convenience for convolution — it also matches how the visual system is organized, which is why frequency keeps coming back (JPEG, image-quality metrics, tone mapping). → see Human perception → Spatial vision / CSF.

• **sidebar — a Fourier transform you can see** (motivation: it's physical, not just an imposed trick): the little **starburst** a distant point of light (a star, a streetlight) makes through a small aperture/lens is ≈ the **Fourier transform of that aperture** (squared magnitude) — round hole → **Airy** disc + rings; stopped-down iris → **sunstar** spikes (one streak ⟂ each blade-edge). The optics performs the transform **for free**, in the instant light crosses the lens — no FFT. Lesson: Fourier is a *natural, physical* way waves + apertures organize themselves; we're borrowing it for images. → ties to [[Compound lenses, and aberration correction]] (diffraction / the **MTF** = FT of the PSF).

We focus on grayscale but it's the same for RGB.

Fourier transform: instead of one value per pixel, one coefficient per sine wave.

It's orthonormal, so coefficients are just dot products (sum/integrals)

• complex exponential carries a cosine + sine per frequency (Euler $e^{i\theta}=\cos\theta+i\sin\theta$); real part = cosine content, imag = sine — needed to capture any phase

• ⚠️ **so many Fourier conventions**: complex-exponential vs real sin/cos form; whether the $2\pi$ lives in the exponent, in the frequency, or in the normalization; continuous transform vs Fourier series vs DFT; normalization ($1/N$, $1/\sqrt N$, or none — and which side carries it); sign of the exponent ("forward vs backward"); angular vs ordinary frequency. These differ across textbooks, NumPy, and signal processing — **always check which convention a source uses** before you trust a formula. [fig-fourier-conventions — author's own "Fourier flavors" convention-warning illustrations, shown **full-width** so each reads clearly; from Fredo's slides 3_Linear_image_processing, slides 67–68 (author-created images)]

Figure: show change of basis matrix [⚠️ legibility: text labels currently unreadable — enlarge / redraw]

In general, Fourier diagonlizes all shift and shift-invariant operators.

• 💡 **Big lesson:** **diagonalize when you can** — express a linear operator in a basis where it's orthogonal/diagonal; **Fourier diagonalizes convolution**, turning it into per-frequency multiplication.

• **sidebar — Fourier and the heat equation** (where this trick came from): Fourier (1807) was solving **heat diffusion** in a 1-D rod with some initial temperature profile. His move was to write the temperature as a sum of **cosines** — and in that basis the heat equation **diagonalizes**: each frequency evolves on its own, and a single cosine simply **decays exponentially in time** (∝ e^{−k²t}), the highest frequencies fastest, so the profile smooths out. The cosines are the **eigenfunctions** of the Laplacian — the same reason Fourier diagonalizes convolution here.

• the heat equation also hands us the **two canonical solution families of linear differential equations: sine waves (in space) and exponentials (in time)** — separation of variables gives **spatial sinusoids** whose **amplitudes decay exponentially**, $\propto e^{-k^2 t}$, the highest frequencies dying fastest, so the profile smooths out. [fig-heat-diffusion-1d — a 1-D temperature profile (amplitude on y, position on x) smoothing over time: shown for (a) a general signal and (b) a single sine wave that simply decays in place]

• the general lesson: **diagonalizing is the win for any linear ODE / PDE** — in the eigenbasis a coupled system decouples into independent scalar equations whose solutions are **exponentials** (e^{λt}) or, for oscillatory / conservative systems, **sine waves**. Find the right basis (the eigenvectors) and a hard differential equation becomes trivial per-mode arithmetic.

• feed a pure sine into any blur → the **same sine**, only scaled and phase-shifted: sines are the **eigenvectors** of convolution, eigenvalue $\hat g(\omega)$ = the kernel's FT [sine-eigenvectors figure]

• blur leaves low freqs untouched, crushes high ones → blur is a **low-pass filter**

• the **convolution theorem** $\mathcal{F}\{I*g\}=\hat I\cdot\hat g$: convolution in space = multiplication in frequency

Give the example of a 2-pixel image visualized as a2D Euclidean plane. Our linear convolution has coefficients 0.8, 0.2 The Fourier transform is just along [1, 1] and [1, -1] (rotation by 45 degrees). Our simple convolution becomes diagonal

• 2-D: two frequency indices (u,v); display as **magnitude** + **phase**

• **symmetries**: a real image has a **conjugate-symmetric** spectrum (half the coefficients are redundant); low freqs at center, high at corners; natural images mostly low-freq; a shift → a **phase ramp**

• **magnitude vs. phase** — how to *read* an image's FT: the **magnitude** shows which frequencies / orientations are present; the **phase** carries the *structure*. Swap two images' magnitude and phase and you recognize the one whose **phase** you kept. [magnitude/phase figure]

• **compact in space ⇔ spread in frequency** — a narrow Gaussian has a wide spectrum; the locality / uncertainty tradeoff that recurs in pyramids & wavelets [compact-space-vs-frequency figure]

• sidebar: **MTF** for lenses: the optical system's frequency response is the FT of its PSF (ties to the diffraction limit, Optics)

• before we lean on it, two honest caveats that the rest of the book keeps bumping into:

• **the DFT treats the signal as cyclic / periodic**: it tiles the image and wraps around, so content leaking off the right edge re-enters on the left → **boundary leakage / ringing**. This is what motivates **windowing** before an FFT, and it is exactly the **periodic-boundary assumption** baked into the FFT deblur solver (forward ref: the FFT-based deconvolution in [[Computational tools]] inherits this wrap-around)

• **the basis has infinite / global support**: every coefficient is a **global wave** spanning the whole image, so there is **no spatial localization** — move a single pixel and *every* coefficient changes. A Fourier coefficient can tell you *which* frequencies are present but not *where*. This is precisely the motivation for **wavelets / pyramids**, which trade a little frequency precision for spatial locality → [[#Pyramids and wavelets]]

• the cyclic-boundary caveat is harmless when we use Fourier to *think* but a hazard when we *measure*: the wrap-around seam injects a spurious **"+"-shaped leakage cross** along the frequency axes that is not in the image. **Windowing** (multiply the patch by a Hann / Hamming / Blackman **taper** that falls to zero at the borders) kills the seam and the cross

• the **tradeoff**: a window suppresses side-lobe leakage but **broadens every spectral peak** (wider main lobe) and discards data near the borders — narrower window = less leakage but worse frequency resolution; Hann / Hamming / Blackman are just different **main-lobe-width vs side-lobe-height** points on that curve

• honest takeaway: we mostly use Fourier for **conceptual understanding** (leakage is harmless), but anyone who **measures** a spectrum — **MTF**, **noise power spectrum**, **periodic-pattern detection** — must window or mistake the leakage cross for real structure [fig-fourier-windowing — interactive: side-by-side 2-D FFT of a test patch (step edge / 2-D sine / sawtooth / box-pulse train) without vs. with a window; the leakage cross appears in the rectangular panel and vanishes in the windowed one, at the cost of broader peaks]

example of spatial aliasing. Show classical sine wave (prudential tower photo downsampling). Intuitively we need to break down the content that cannot be represented (return gray instead of black or white). You can think of it as the average over a pixel. More formally, we need Fourier.

As it turns out, sampling (and so aliasing) is **easy in Fourier — because Fourier is full of shifts.** Sampling is multiplication by a **Dirac comb** in space, which by the convolution theorem is a **convolution in Fourier**. That sounds harder, not easier — *but the comb has an especially simple spectrum*: the Fourier transform of a Dirac comb is **another train of Diracs** (the comb is self-dual). And convolving by a Dirac is just a **shift**, so convolving by a *train* of Diracs is just a **sum of shifted copies** — the spectrum gets **replicated** at every multiple of the sampling rate, no smearing, pure translation. So the messy-sounding "convolution" is, here, nothing but shifts — which is the whole reason sampling is so legible in the frequency domain (and the replicas are exactly what alias when they overlap; developed in [[#Sampling, downsampling, and aliasing]]).

• **Nyquist–Shannon**: to represent a signal you must sample at **more than twice** its highest frequency; above that, high frequencies **fold (alias)** down to low ones — the moiré / wagon-wheel effect [aliasing figure] [📺 embed aliasing/sampling explainer → https://www.youtube.com/watch?v=yr3ngmRuGUc, see [[Video Resources]]]

• anti-aliasing: low-pass (pre-filter) before you downsample

• **sidebar — Aliasing on purpose: dynamic moiré prints**: aliasing / moiré is usually a nuisance, but it can be **engineered**. A printed **micro-pattern** photographed by a camera produces a moiré whose appearance **encodes information** — the relative camera pose / motion — because the beat pattern is exquisitely sensitive to the sampling geometry. Cite MIT's **KinéCam** dynamic prints (https://hcie.csail.mit.edu/research/kinecam/kinecam.html). Ties to **structured / coded patterns** elsewhere in the book.

• the space-vs-frequency tradeoff (sharp in one ⇒ spread in the other)

• sidebar: a primal-domain (no-Fourier) perspective too

• perfect sinc vs practical filters. (more in resampling section)

We will discuss more in later chapter, but quickly show:

sharp input image

blurry image (say with box. Maybe just 1D ?) plus noise.

linear inversion with amplified noise.

Fourier MTF

domain operations, function from plane to itself

start with example of scaling up (upsampling). Missing values. Say downsampling will be a bit different (too many values) but the solution will be surprisingly similar.

start naive : loop over input pixels. Point out problem (gaps)

better solution: loop over output (inverse mapping: for each output pixel, inverse-transform and sample the input)

• where from vs where to: always inverse transform

challenge

derived with intuitive 1D linear interpolation

say in2D it can't be purely linear unless we triangulate

sidebar: smart triangulation where we pick the best diagonal for each square (the one joining the two most similar values)

Say we do linear horizontally first, then vertical

exercise : show it's the same if we do it in the opposite order.

• pseudocode: upsample = loop over output, inverse-map to fractional coords, bilinear-blend the four corners (two horizontal blends then one vertical)

say that the interpolation perspective is fine for linear and upsampling, but because this is about (re)sampling, Fourier has better insights. In practice, means reconstruction through convolution.

side bar: proper resampling analysis: start with sampled signal, reconstruct, prefilter, resample. The final filter is a combination of prefilter and reconstruct. Depending on whether we up or downsample one of them dominates.

• warning: much about bicubic wrong on the internet. Including in old python libraries

They all approximate a perfect sinc but with a finite support and try to find a tradeoff between loss of medium frequency and aliasing of high frequencies. Show in Fourier : perfect box, smooth approximation

Netravali Mitchell.

form of bicubic.

required symmetries, remaining two parameters

perceptual study.

point out that photoshop gives you a choice.

The key is how you scale the filter.

• 💡 **Big lesson:** **prefilter before you downsample** — when you throw samples away, first blur / area-average out the detail too fine for the new grid, or it **aliases** into bold false low-frequency moiré; naive nearest-pixel decimation is the classic bug (→ **L16**)

• (Andrew Adams' **rotation-challenge** sidebar + `fig-rotation-challenge` + the interactive `fig-rotation-resample-interactive` now sit **before** this section — they cap the reconstruction-filter-quality discussion)

• for a **shear / rotation / homography** the ideal prefilter is **spatially varying and anisotropic** — an output pixel back-projects to a stretched **ellipse** in the input, so exact filtering is costly; **mip-maps** and **EWA** (elliptical weighted average) approximate it

• the linear, **isotropic** filters so far reconstruct only inside the **Shannon/Nyquist window**; with no extra info that's all you can get

• but natural images have **regularities** (→ [[Imaging as Bayesian inference with probabilistic priors]] image-manifold prior) — notably **local anisotropy**: content lies mostly **orthogonal to edges**, so the local spectrum is a **line/wedge** and the **aliases don't fill** the Fourier plane

• so we can **accept aliasing** and **steer the reconstruction** along the edge (wide along, narrow across) — equivalently, in Fourier, reconstruct only where the spectrum is and ignore the alias-free regions. Reconstruction becomes **non-linear & data-driven** (orientation must be estimated from the low-res data)

• the workhorse of **demosaicking** and older **super-resolution**: two steps — **analyze local orientation/pattern, then steer a reconstruction filter**

• **modern version**: **learn** the regularities of natural images and how to reconstruct them (→ learning chapters)

• `fig-sampling-steered-recon` [interactive] — a steered variant of `fig-sampling-2d-pipeline`: edge sampled w/o prefilter (aliases), reconstructed with an **oriented elliptical sharp-bicubic** filter (user sets orientation θ + scale along/across); the oriented passband hugs the signal's spectral line and skips the aliases → edge reconstructs; misalign → aliasing returns

• recap: resampling = reconstruct-then-resample, reconstruction = convolution with a finite-budget sinc approximation

• forward: the reduce step → Gaussian/Laplacian pyramids (next); inverse-mapping → warping & morphing; resampler quality → image metrics

• a multiresolution representation — **halfway between primal (space) and Fourier (frequency)**: localized in *both*, which is exactly why it's so practical

• multiresolution, coarse-to-fine (original papers, 1980s)

• repeatedly **blur then downsample by 2** → a tower of ever-coarser, smaller images (each ¼ the pixels): the **reduce** operator

• cheap (a geometric series → only ~4/3 the original pixel count in total); the basis for coarse-to-fine search and **scale space** [Gaussian pyramid figure]

• construction: each level = **G_k − expand(G_{k+1})** (the detail the coarser level missed) — a **band-pass** image [Laplacian pyramid figure]

• reconstruction: **exactly invertible** — add the levels back coarse → fine (an **encoder → decoder**) [encode/decode figure]

• pseudocode: Laplacian build (reduce + difference loop) and collapse (expand + add loop), each the exact inverse of the other

• frequency intuition: each level holds a **band** of detail, coarse → fine [frequency-bands figure]

• properties: **decorrelated** and **sparse** on natural images (mostly near-zero → compresses, denoises) — sparsity drives compression (JPEG 2000) and denoising (coring)

• **overcomplete** (>1 coefficient per pixel — a frame, not a tight basis); only **approximately shift-invariant**; **isotropic** (no orientation) — each limitation motivates a relative

• **wavelets**: a true tight basis (one coefficient per pixel, no redundancy) → natural for compression (JPEG 2000); but less Fourier-friendly, not shift/rotation-invariant

• sidebar: **Fourier, wavelets, and the locality dial** — Fourier (all frequency) ↔ primal grid (all space); wavelets/pyramids in between, a tunable space-vs-frequency tradeoff

• **steerable pyramids** (Simoncelli & Freeman 1995): add **orientation** (oriented sub-bands), nicely shift/rotation-invariant, even more overcomplete — texture, video magnification; (**multigrid** in passing)

• multiscale / **pyramid blending**: blend each Laplacian level under a **Gaussian-blurred mask**, recombine → seamless joins (the apple/orange) — see compositing & pano [pyramid blending figure]

• exposure fusion, HDR+ tone mapping — see HDR

• pyramid **denoising / coring**: zero out the small detail coefficients (mostly noise), keep the large ones — see Denoising [coring figure]

• local Laplacian filtering (edge-aware detail) — see edge-preserving

• coarse-to-fine (optical flow), scale (SIFT), pooling / U-Net (deep nets) — forward refs

• recap [[#Image measurements as integrals over the plenoptic function]]: a pixel value is an integral of the plenoptic function against a sampling kernel. Now stack everything: the **scene** becomes a vector $\mathbf{x}$, the **image** a vector $\mathbf{y}$, and the imaging system is a **matrix** $A$ — $\mathbf{y}=A\mathbf{x}+\mathbf{n}$. Each **row** of $A$ = one pixel's kernel (which scene elements it sums, with what weights).

• the big-lesson tie-in: **image formation is selecting and combining rays** — the pinhole *selects* a single ray per pixel, the lens *combines* a cone; $A$ is exactly that selection/combination, written down. (→ [[Big Lessons]]; [[Pinhole image formation and linear perspective]]; [[Lens image formation]].)

• **flatland**: a 2-D world (a 1-D scene, a 1-D sensor), both **discretized**, so $A$ is a small drawable matrix. Two standard ways to feed $A$: a **billboard** (next section) or **ray space** (the section after).

• input = a **flat 2-D billboard** at a chosen depth — the scene as a 1-D row of colored points; $A$ maps billboard → sensor. Show each system's flatland $A$ as a heat-map [`fig-linsys-flatland`, `fig-linsys-matrices`]:

• **pinhole** — each sensor pixel sees **one** billboard point → $A\approx$ a permutation/identity (sharp; throws away nearly all light).

• **pinspeck** (anti-pinhole / accidental occluder) — the **complement** of a pinhole: $A=$ all-ones minus the pinhole row pattern (Cohen's accidental cameras).

• **lens** — each pixel integrates a **range** of billboard points (the defocus disc off the focal plane): $A=$ a **banded / blur** matrix; $\approx$ identity at the focal depth, wider off it.

• **corner camera** — an occluding edge turns the scene into a **cumulative integral**: $A\approx$ lower-triangular; differentiating the measurement recovers angle (penumbra / accidental cameras).

• **stereo pair** — two pinholes → two images stacked: $A$ is **taller** (more rows); the two viewpoints give disparity → depth.

• **coded aperture** — a patterned mask in the aperture: $A=$ a **coded blur** engineered to be broadband / zero-free (invertible) and depth-revealing (**forward-ref** [[Computational optics and coded imaging|Coded aperture]]).

• **the limitation**: the billboard $A$ **depends on depth** — move the billboard and $A$ changes (the lens blur widens, the stereo disparity shifts). One matrix describes only one depth. That is exactly what the ray-space view fixes.

• input = the **light field**: the set of **all rays** arriving at the system (ray space — [[Light fields and plenoptic cameras]]). The imaging system is then a **fixed** linear projection of ray space → image, **independent of scene depth** (depth becomes structure *within* $\mathbf{x}$, not a change to $A$).

• the **same cameras** re-expressed as projections of ray space: a pinhole = a thin slice; a lens = an integral along a sheared line; a coded aperture = a coded projection; a stereo / light-field camera = **more rows** (more of ray space measured). [`fig-camera-tradeoffs-1`, after [@levin-etal-2008-lightfield]]

• cite [@torralba-etal-fcv] (the camera as a linear system) and [@levin-etal-2008-lightfield] (trade-offs via light-field projections).

• the idea to land: **much of computational photography and computational optics is the search for new linear imaging systems $A$ — with matching reconstruction methods — that extract better images and extra information, notably depth.**

• extend $\mathbf{x}$ with **time**: a moving scene under an exposure is again a linear system, now over **space–time**. The exposure integrates along the motion trajectory (motion blur = a linear operator whose kernel is the path).

• **motion-invariant photography** ([@levin-etal-2008-motion]): sweep the camera along a **parabolic** trajectory during the exposure so the blur PSF is the **same for every object speed** → a single deconvolution deblurs all of them. Designing the *operator* (the parabolic sweep) turns an otherwise object-dependent $A$ into one invertible kernel. [`fig-linsys-motion-invariant`]

• the temporal sibling of coded aperture / wavefront coding — *engineer the operator so it inverts cleanly*; full treatment in [[Computational optics and coded imaging]] and [[Motion blur, temporal sampling, and resampling]].

• the clean $\mathbf{y}=A\mathbf{x}$ is an idealization; a few real departures matter.

• **clipping / saturation** — wells and code values have ceilings; past saturation the response flatlines. A hard nonlinearity that **destroys information** (no inverse) — poisons deconvolution and exposure merges, so saturated pixels must be detected and handled (→ [[Blind deblurring]], [[HDR merging]]).

• **gamma & quantization** — encoding nonlinearities, both benign. **Gamma** is an *invertible* power law $V=L^{1/\gamma}$: decode to work in linear light, or operate directly on encoded values — exposure becomes $V\mapsto 2^{EV/\gamma}\,V$ and a white-balance gain $g_c$ becomes $V_c\mapsto g_c^{1/\gamma}\,V_c$ (exact for a pure power law; ≈ for sRGB's linear toe). **Quantization** is best **modeled as additive noise** (uniform, variance $\Delta^2/12$ for a code step $\Delta$), dwarfed by photon/read noise at any sane bit depth — it matters only where it compounds (repeated 8-bit round-trips, hard tone curves → banding, the cue for dithering). (→ [[Measuring and encoding color|Color technology]].)

• **not everything we want is linear in the scene** — depth, motion, material, identity are nonlinear read-outs of the light, not matrix inverses (→ [[Photographic representation and its limits]]).

• **a linear forward model ≠ a linear solution** — recovery leans on priors/inductive biases, so the map from measurement to estimate is thoroughly nonlinear. *Linear physics, nonlinear inference.*

• the question that organizes everything after this chapter: **how invertible is $A$?** — how well can you recover $\mathbf{x}$ (a sharp image, a depth map, more) from $\mathbf{y}=A\mathbf{x}+\mathbf{n}$?

• two complementary ways to get **more invertible information**:

• **coded imaging / computational optics** ([[Computational optics and coded imaging]]): keep the measurement count but **redesign $A$ to be better-conditioned** — coded aperture, wavefront coding, the **lattice-focal lens** ([@levin-etal-2009-4dfreq]) — so the same data inverts more cleanly (and reveals depth).

• **light fields & multiple-exposure photography** ([[Light fields and plenoptic cameras]], [[Multiple exposure imaging]]): **make $A$ taller** — take *more* measurements (more views, more exposures) — so the system is more determined and carries more information.

• the tool for *reasoning* about invertibility is a **change of basis that diagonalizes $A$** — most often **Fourier analysis** (convolution → per-frequency multiplication), which makes the conditioning (where $A$ has small / vanishing singular values) and the inversion legible. (→ [[Linearity, Fourier, Aliasing and deblurring]]; [@levin-etal-2009-4dfreq] do exactly this 4-D-frequency analysis for computational cameras.)