12.6 Deep learning approaches to sparse matching🔗⧉

The classical pipeline, a hand-designed detector, a hand-designed descriptor, and a nearest-neighbor matcher, is exactly the kind of three-stage hand-built system that deep learning tends to replace end to end. Each stage was an act of human intuition: Harris Harris & Stephens 1988 decided a corner was where the structure tensor had two large eigenvalues; SIFT Lowe 2004 decided a neighborhood was best summarized by gradient-orientation histograms; the matcher decided two points correspond when their descriptors are nearest in Euclidean distance. Every one of those decisions is a candidate for the L7.6show move: replace the hand-designed operator with one trained from data, and the history of learned matching is the story of that move being applied to more of the pipeline.

It arrives in three waves, each absorbing more of the pipeline. First, learned local features that play SIFT's role: a single network detects and describes, trained for repeatability and matchability rather than designed by intuition (LIFT Yi et al. 2016 (LIFT), SuperPoint, D2-Net Dusmanu et al. 2019 (D2-Net), R2D2 Revaud et al. 2019 (R2D2)). Second, learned matching that reasons about the whole set of features at once instead of deciding each point independently: SuperGlue and the faster LightGlue Lindenberger, Sarlin & Pollefeys 2023 (LightGlue) cast matching as an attention-based assignment, while detector-free matchers like LoFTR skip keypoints entirely. Third, the 3-D-aware turn, where DUSt3R and MASt3R Leroy et al. 2024 (MASt3R) regress dense pointmaps straight from an image pair, so correspondence, camera pose, and depth all fall out together and matching stops being a separate stage at all. Front end, then middle, then the whole thing folded into geometry. The task is fixed; the pipeline is progressively learned.

12.6.1 Learned local features: SIFT's job, done by a network🔗⧉

The first wave replaces SIFT's two separate stages, the Difference-of-Gaussian detector and the gradient-histogram descriptor, with a single convolutional network that does both. The architecture is now standard: a shared encoder processes the image once, then splits into two heads. A detector head emits a keypoint score map (where the keypoints are), and a descriptor head emits a dense descriptor field (one descriptor vector per location). The two heads are trained together, so the points the detector likes are exactly the points the descriptor can match well, a cooperation the hand-built pipeline never had, since SIFT's detector and descriptor were designed independently. LIFT (Yi et al., 2016) first showed the whole detect-orient-describe pipeline could be learned end-to-end; SuperPoint (DeTone et al., 2018) DeTone et al. 2018 (SuperPoint) made it fast and self-supervised; D2-Net (Dusmanu et al., 2019) and R2D2 (Revaud et al., 2019) refined what "a good keypoint" should mean (Figure 12.6.1⛶ fullscreen).

The obvious difficulty is supervision: there is no ground-truth "corner" to train against. Nobody can label, pixel by pixel, which points are truly repeatable across viewpoint and lighting, so a network has nothing to imitate. SuperPoint's answer, homographic adaptation, became the template for later work. Take an image, warp it by many known random homographies, run the current detector on each warped copy, then un-warp the detections back to the original frame and demand that they agree. A point that fires consistently across many synthetic warps is, by construction, repeatable under those transforms; a point that fires in only one warp is noise. Repeatability-under-known-transforms becomes the training signal, manufactured from the warps themselves, with no human annotation. R2D2 pushes the idea further by training the descriptor for reliability alongside the detector's repeatability, explicitly learning that a point can be easy to find yet hard to match (a repeated texture), and that those points should be down-weighted.

Little about the interface changes. The task is identical to SIFT's (emit repeatable, matchable points), and only the operators move from hand-design to data: what counts as a keypoint, and what the descriptor encodes. Because the output is still a set of (location, descriptor) pairs, these networks are drop-in front ends: their points feed the very same ratio-test-then-RANSAC back end from Chapter 12.5, Robustness: the ratio test and RANSAC, with no change downstream. That drop-in property is why they were adopted quickly: you could swap SuperPoint for SIFT in an existing structure-from-motion pipeline and keep everything else.

12.6.2 Learned matching: reason about the whole set at once🔗⧉

Even with learned features, the matching step in the classical pipeline is strictly local: each keypoint in image A is matched to the keypoint in image B whose descriptor is nearest, $\arg\min_j \lVert d_i - d_j \rVert$, decided independently for every point. This is blind to a fact from the geometry: the matches must be mutually consistent. Correspondences carry geometric context: nearby points often share a surface or motion, and an assignment must account for unmatched points. General scenes do not obey one global image transform, and drawn match lines can cross even when the matches are correct. Nearest-neighbor matching throws all of that away and decides each point in isolation, then leans entirely on RANSAC downstream to clean up the mess. The second wave asks the matcher itself to reason about the whole set at once.

SuperGlue (Sarlin et al., 2020) Sarlin et al. 2020 (SuperGlue) does this with attention. Treat the two keypoint sets as the nodes of a graph and alternate two kinds of attention: self-attention within an image (so each point is informed by the other points in its own image: the local geometric context) and cross-attention between images (so each point is informed by the candidate matches in the other image). Stacked several layers deep, this lets every match be decided in light of all the others: a point near an unambiguous corner inherits confidence from its neighbor, an isolated point on repeated texture is rightly distrusted. The network outputs a learned score matrix $S$, where $S_{ij}$ scores keypoint $i$ in A against keypoint $j$ in B, and the final assignment is found by solving a differentiable optimal-transport problem: find the soft assignment $P$ that

For SuperGlue, the scores are augmented with a dustbin row and column for unmatched points. Entropy-regularized transport is solved with prescribed marginals; these are not all unit marginals on an arbitrary rectangular matrix. The dustbin accounts for unequal set sizes. Sinkhorn iterations produce a soft assignment, followed by mutual-selection and confidence tests. See the authors’ implementation.

The cost of that global reasoning is speed, because the attention layers are quadratic in the number of keypoints, which is painful on a phone or a robot. LightGlue (Lindenberger et al., 2023) keeps SuperGlue's quality but makes it adaptive: it predicts after each layer whether the match is already confident enough to stop, and it prunes keypoints that are clearly unmatchable early, so easy image pairs finish in a few layers and only the hard ones pay full price. The result is a matcher that runs at interactive rates on-device while inheriting SuperGlue's global-consistency benefit.

A more radical move drops the detector entirely. LoFTR (Sun et al., 2021) Sun et al. 2021 (LoFTR) is detector-free: instead of finding keypoints and then matching them, it matches densely and coarse-to-fine, using transformers to establish correspondences on a coarse grid and then refining each to sub-pixel accuracy. The gain is largest on the surfaces where detectors fail worst (low-texture walls, repetitive patterns, weakly-textured indoor scenes), where Harris or SuperPoint fire almost nowhere, leaving classical matching with too few points to fit a geometry. By never committing to a sparse keypoint set, LoFTR can produce correspondences where there are no distinctive corners at all. In this setup, the detector is optional.

Sinkhorn normalization alternates row and column rescaling toward the prescribed marginals. Those differentiable operations let the assignment loss train the score-producing network. The unmatched entry is essential: a point outside the overlap or behind an occluder should not be forced into a plausible-looking pair merely to complete the assignment.

12.6.3 The 3-D-aware turn: pointmaps subsume matching🔗⧉

The third wave stops treating matching as a problem in its own right. The observation is that in nearly every application, sparse matching is only a means to an end, since you match features in order to estimate the camera pose, or to triangulate 3-D structure, or to build a panorama. The classical chain is long and brittle: detect, describe, match, ratio-test, RANSAC the fundamental or essential matrix, recover relative pose, triangulate. Each stage can fail, and errors compound. Instead, ask a network to predict the 3-D geometry directly, and derive matching from that prediction.

DUSt3R predicts paired pointmaps in a shared camera coordinate frame Wang et al. 2024 (DUSt3R). Nearby predicted 3-D points can support correspondence, but geometry is noisy and may be ambiguous in scale or visibility. Camera parameters and multi-image consistency can require additional estimation or alignment. MASt3R adds descriptors trained for matching; the official project describes its matching and alignment stages. These outputs complement geometric validation rather than eliminating it.

Pointmap regression changes where geometric information enters the pipeline. It does not make every predicted point a measured scene location. Part 36, 3D and depth develops reconstruction; here evaluate the correspondence accuracy, overlap and downstream geometric residuals needed by the application.

A pointmap associates an image location with a predicted three-dimensional coordinate. Expressing two maps in a shared frame makes their geometry directly comparable, instead of first requiring a detector and a list of descriptor matches. That is the educational shift: geometry can help propose correspondence, as well as be reconstructed from correspondence. It moves the difficult inference into the network and subsequent alignment; it does not remove that inference.