💡 **Big lesson (L4.8 here · diagonalize when you can):** a pure translation is *diagonal* in the Fourier basis — it leaves every magnitude untouched and tilts only the phases by a ramp whose slope **is** the shift. So a brute-force $O(\text{window}^2)$ shift search collapses to a single FFT pair: form the normalized cross-power spectrum, inverse-transform, and read the shift off the location of the resulting delta. Whenever the operator is a shift, a convolution, or a correlation, ask first whether the frequency domain turns the work into a per-frequency (diagonal) operation. *(L4.8, first placed in [[Linearity, Fourier, aliasing and deblurring]]; here it turns a search into a transform.)*

The previous part transported pixels along maps that were *given* or *hand-drawn*. This part *estimates* the maps, and the cheapest, most robust way to start is not to match every pixel but to match a **sparse** handful of points you can find again. Three stages, in order: **detect** distinctive points (the *where*), **describe** each one's neighborhood (the *what*), and **match** descriptors across images (the *which-goes-with-which*). Detect for **repeatability** (the same physical point fires in both images, despite viewpoint, scale, and lighting), describe for **distinctiveness** (the descriptor is unique enough to match unambiguously and invariant enough to survive the change between images). The raw matches are then filtered — ratio test and RANSAC — in the next section.

💡 **Big lesson (the structure tensor, three jobs):** the same $2\times2$ matrix $M=\sum I\nabla I\nabla I^{\top}$ answers three questions across this part. *Is this a good point to detect?* — yes where $M$ has two large eigenvalues (a corner): **Harris / Shi–Tomasi**, here. *Is this patch's motion solve well-conditioned?* — same matrix invertible: **KLT** ([[Feature tracking]]). *What is the local flow constraint?* — $M$ is the normal-equation matrix $A^\top A$: **Lucas–Kanade** ([[Optical flow]]). Detect with it, track with it, solve flow with it — one matrix, read three ways. Corners are special precisely because the *same* both-directions-vary condition that localizes a point also makes its motion recoverable.

The previous section estimated a **dense** flow field — a motion vector at *every* pixel — between *one* pair of frames. This section does the complementary thing: follow a **sparse** handful of *distinctive* points across *many* frames. It is the same physics (brightness constancy, the aperture problem) restricted to the places where the math is well-behaved, and run forward in time. The payoff of the restriction is the chapter's spine: **the matrix that says a patch is a good corner to *detect* is the same matrix that must be invertible to *track* it.** Detect with it (Harris), track with it (KLT), refuse to track where it is singular (Shi–Tomasi). One $2\times2$ matrix, three jobs.

💡 **Big lesson (recurrence of L5.3 — a learned operator swaps a hand-designed prior for one learned from data):** classical KLT tracks a hand-built brightness patch by a hand-derived least-squares update; modern point trackers (PIPs, CoTracker) keep the *exact same task* — "where did this point go?" — but replace the patch with **learned features** and the greedy per-point update with a **learned, occlusion-aware, track-together** module. The skeleton (initialize a trajectory, iteratively refine it against appearance, predict visibility) is unchanged; only the operator is now fit to data. (→ see Big lesson **L5.3**; first appears in [[Deep learning]]. The same "neuralise the classical iteration" move powers RAFT in [[Optical flow]].)

A keypoint matcher hands you a pile of correspondences that is **mostly noise**: repeated texture, occlusion, and ordinary descriptor collisions all produce confident-but-wrong matches, and a single wrong match wrecks a least-squares fit (one bad point pulls the whole model). Two classic, complementary filters make matching usable — one prunes *before* fitting, the other fits *despite* the survivors' remaining outliers.

The classical pipeline — hand-designed detector, descriptor, nearest-neighbor matcher — is exactly the kind of three-stage hand-built system deep learning likes to swallow whole (the **L5.3** move: learn the operator from data). It arrives in three waves. First, **learned local features** that play SIFT's role — joint detector-and-descriptor networks (**LIFT**, **SuperPoint**, **D2-Net**, **R2D2**) trained for repeatability and matchability, often **self-supervised** by warping an image with known homographies and demanding consistent keypoints. Second, **learned matching** that reasons about the *whole set* of features at once instead of independent nearest-neighbors: **SuperGlue** (and the faster **LightGlue**) cast matching as graph/attention-based assignment (an optimal-transport-flavoured solve), while **detector-free** matchers like **LoFTR** skip keypoints entirely and match dense coarse-to-fine. Third, the **3-D-aware** turn — **DUSt3R / MASt3R** regress dense **pointmaps** directly from an image pair, so correspondence, camera pose, and depth fall out together with no explicit keypoint stage at all; matching stops being a separate step and becomes a by-product of predicting geometry. Each wave keeps the *task* and learns more of the *pipeline*.

Once you have descriptors, the cost is dominated not by computing them but by the **nearest-neighbor search** in a high-dimensional space — and exact NN is hopeless at scale (the curse of dimensionality defeats kd-trees past a few dozen dimensions). So the toolkit is **approximate** NN and clever propagation. For sparse descriptors: randomized **kd-trees / FLANN**, **locality-sensitive hashing** (collide nearby points into the same bucket), and **product quantization** for billion-scale retrieval (the lineage that became FAISS / ScaNN vector databases). For *dense* patch correspondence, **PatchMatch** abandons search trees entirely — random initialization plus **propagation** (good matches are spatially coherent, so a neighbor's offset is a great guess) plus occasional random refinement converges astonishingly fast ([[Patch match]]). Worth singling out is **Andrew Adams's random-projection lineage** — Gaussian kd-trees and the **permutohedral lattice** — which finds good high-dimensional matches by projecting onto a few random directions and splatting/propagating on a lattice; the same machinery accelerates the bilateral filter and NL-means denoising ([[Slopendium/Denoising]]). The recurring lesson: at scale, *good-enough-and-fast* beats *optimal-and-slow*, and **propagation** (exploit coherence) often beats **search** (treat each query independently).

💡 **Big lesson (the chapter's core — *brightness constancy + the aperture problem*):** track a point by the one thing that's (assumed) conserved — its **brightness**. Linearizing that single scalar equation gives **one constraint per pixel**, $I_x u + I_y v + I_t = 0$, for **two** unknowns $(u,v)$ — so locally motion is **fundamentally underdetermined**: you can only see the component **along the image gradient** (normal flow), never the component **along an edge** (the *aperture problem*). Every flow method is a strategy for supplying the missing second equation — from a **neighbor** (Lucas–Kanade: pixels in a patch share a motion → the well-/ill-posedness is the **same corner/edge/flat structure tensor as Harris**), from a **smoothness prior** (Horn–Schunck), or from **learned context** (RAFT). *Only the gradient carries motion information, and one gradient is never enough.*

The learned-flow story is best told as a single pattern rather than a parade of networks: **take the classical iteration of [[Optical flow]] and *unroll* it into a differentiable network trained end-to-end**, so the hand-tuned data term, smoothness regularizer, and coarse-to-fine schedule become learned. **FlowNet** first showed a CNN could regress dense flow at all. **PWC-Net** made the lineage explicit by folding the *classical* pipeline — **P**yramid, **W**arping, **C**ost-volume — into a compact, fast network: the same coarse-to-fine warping of the previous section, now with learned features and a learned update. **RAFT** is the design that came to dominate — build a 4-D **all-pairs correlation volume** (every pixel's similarity to every other) and apply a **recurrent update operator** that behaves like an unrolled iterative optimizer refining the field; it is the flow analog of "neuralize the classical solver" and the reason RAFT-style modules now appear in stereo, scene flow, and tracking. Transformer flows (**FlowFormer**, **GMFlow**) reframe the same problem as **global matching**, replacing the local cost-volume update with attention. Throughout, the *skeleton* — features, a cost/correlation volume, an iterative update, coarse-to-fine — is the classical one; learning supplies the operators.