20.1 Manual panorama stitching from multiple views🔗⧉

Overlapping photographs can cover a scene wider than one frame. Stitching requires a map between their common content. For an ideal pinhole camera rotating about a fixed projection center, or for points on one plane, that map is a homography. Start by clicking correspondences; Chapter 20.2, Automatic panorama stitching from multiple views and feature matching replaces those clicks with feature matches.

20.1.1 The scenario, and the one rule: rotate, don't translate🔗⧉

Here is the setup. You stand in one place and pan the camera across a scene, taking overlapping frames. Choose one frame as the reference; reproject, or warp, the others into its coordinate system; blend the overlaps. The output is a virtual wide-angle view assembled from shots no single exposure could have captured (Figure 20.1.1⛶ fullscreen).

For a general scene, keep the camera's effective projection center approximately fixed as it turns. The ideal rotation model then relates visible scene points independently of depth. In practice, lens distortion, rolling shutter and viewpoint shifts can violate it. A panorama head adjusted around the no-parallax position reduces viewpoint shifts.

Translation creates depth-dependent parallax: a nearby railing shifts relative to a distant building. One homography generally cannot align both. Local warps or depth-aware reconstruction can address some of the discrepancy; translation does not make every possible two-dimensional correspondence map impossible. A planar scene remains an important homography case.

One pointer, in keeping with our standing division of labor. The warping itself (forward vs. inverse warp, resampling, antialiasing) and the closely related perspective-distortion correction are developed in Matching pixels across space and time. Here we need only the geometry (which $H$ relates the views) and how to solve for it.

20.1.2 Refresher: pinhole projection is "divide by depth"🔗⧉

Everything below is a consequence of one fact about the pinhole camera, so let us refresh it. A 3D point $(x,y,z)$ in the camera's frame images to the point $(x/z,\,y/z)$ on the plane $z=1$. Projection is division by depth. This is why far things look small: a larger $z$ shrinks $(x/z,y/z)$ toward the center. The whole nonlinearity of perspective lives in that single divide.

Define $R$ to transform a ray expressed in camera 1 coordinates into camera 2 coordinates. If the camera-to-world orientations are $Q_1,Q_2$, then $R=Q_2^{\mathsf T}Q_1$. This convention avoids confusing a physical camera rotation with the inverse coordinate transform.

We know the ray from each source pixel, but not the depth of the scene point along it. Under pure rotation, all positive depths along that ray share the same destination direction.

20.1.3 Why you don't need 3D: depth cancels for a pure rotation🔗⧉

The idea is best stated as an observation about a single viewing ray: all points along one viewing ray reproject to the same place. A pixel $(x,y)$ in image 1 is the image of an entire ray of 3D points $(tx,\,ty,\,t)$, one for every depth $t$, all sharing the same direction. Rotate that whole ray by $R$ and re-project. The rotation scales linearly, so the rotated ray is $R\,(tx,ty,t)^\top = t\,R\,(x,y,1)^\top$: the depth $t$ is a common factor of all three coordinates. But projection divides by the third coordinate, and dividing kills that common factor. Every point on the ray, at every depth, lands on the exact same pixel in the rotated view. The destination pixel does not depend on depth at all (Figure 20.1.3⛶ fullscreen).

We may therefore represent the source ray at any convenient nonzero scale. Rotating it and dividing by its third coordinate gives the same destination pixel regardless of the chosen depth.

Chaining the three steps, un-project (apply $K^{-1}$ to go from pixels to rays), rotate (apply $R$), re-project (apply $K$ to go from rays back to pixels), gives the view-to-view map

Here $K_1,K_2$ are the two intrinsics matrices and $R$ follows the convention above. With equal intrinsics this becomes $KRK^{-1}$; with normalized image coordinates it becomes $R$. The relation holds for rays with a valid destination projection, not for all output locations regardless of visibility.

The pure-rotation model avoids estimating scene depth. Fitting and rendering it still require numerical care, and geometric error minimization is nonlinear even though an algebraic initialization can be obtained linearly.

20.1.4 Homographies and homogeneous coordinates🔗⧉

The map $\mathbf{x}'\simeq H\mathbf{x}$ deserves its own vocabulary, because the same object recurs throughout the rest of the part. We have been implicitly using homogeneous coordinates: represent a 2D point $(u,v)$ by three numbers $(x,y,w)$, with $u=x/w$ and $v=y/w$. To recover the ordinary Euclidean point you divide by the last coordinate. A point therefore has no unique triple: every scaling $(wx,wy,w)$ names the same point, so homogeneous coordinates are defined up to scale. Two payoffs justify the bookkeeping. First, they make perspective (that very divide) and translation expressible as a single matrix multiply. Second, they are literally "treat the image as a plane at $z=1$ in 3D," which is exactly the picture from the previous section.

A homography is represented by an invertible $3\times3$ matrix up to nonzero scale. It preserves straight lines, but can send points to infinity where the homogeneous denominator vanishes. Not every arbitrary corner arrangement gives a finite, nonsingular warp of the entire source rectangle.

How much freedom does $H$ have? It has nine entries, but it is defined up to scale, since $H$ and $kH$ produce the same map, because scaling all of $(w'x',\,w'y',\,w')$ by $k$ leaves the Euclidean point $(x',y')$ unchanged. So a homography has 8 degrees of freedom, not 9. It tells us exactly how much data we need: 4 point correspondences supply 8 numbers, just enough to pin $H$ down. (This is the same $s=4$ minimal sample random sample consensus (RANSAC) draws in the next chapter, Chapter 20.2, Automatic panorama stitching from multiple views and feature matching.)

Writing out $H=\left(\begin{smallmatrix}a&b&c\\ d&e&f\\ g&h&i\end{smallmatrix}\right)$, the per-point formula is

The shared denominator $w'=gx+hy+i$ is the perspective divide; the entries $g$ and $h$ are what make parallel lines converge. Setting $g=h=0$ (and $i\neq0$) makes the denominator constant, which removes the perspective entirely and leaves a plain affine map. This standard projective-geometry material is laid out at length in Hartley and Zisserman and Szeliski.

20.1.5 Solving for $H$ from correspondences🔗⧉

The goal is concrete: given $\geq 4$ matched point pairs $\mathbf{x}_i \leftrightarrow \mathbf{x}_i'$, find the homography $H$ that maps each $\mathbf{x}_i$ to $\mathbf{x}_i'$. In manual stitching the user supplies these pairs by clicking matching landmarks across the overlap; automatically they come from feature matching (next chapter).

At first glance $\mathbf{x}'\simeq H\mathbf{x}$ looks nonlinear, because of the divide. The trick is to clear the denominator by cross-multiplying. Take $x' = (ax+by+c)/(gx+hy+i)$, multiply both sides by the denominator, and you get $ax+by+c-x'(gx+hy+i)=0$, a single equation that is now linear in the unknown entries of $H$. Each correspondence yields two such equations (one from $x'$, one from $y'$). Stack them into

where $\mathbf{h}$ is the 9-vector of $H$'s entries and $A$ has two rows per correspondence. Four pairs give 8 equations.

The all-zero vector trivially solves the homogeneous equations and cannot represent a homography. Direct linear transformation (DLT) instead minimizes $\lVert A\mathbf h\rVert^2$ subject to $\lVert\mathbf h\rVert=1$, using a right singular vector. Before fitting, normalize coordinates in each image to zero centroid and mean distance $\sqrt2$, then undo the normalizations.

For a normalized correspondence $(x,y)\mapsto(u,v)$, add the two rows

to $A$. If $T_1,T_2$ normalize source and destination coordinates, denormalize with $H=T_2^{-1}H_nT_1$.

assert number_of_pairs >= 4
reject_repeated_or_nearly_collinear_configurations(pairs)
source, T1 = normalize_points(source_points)
target, T2 = normalize_points(target_points)
A = dlt_rows(source, target)
h = full_right_null_vector(A)  # retain the ninth vector for an 8-by-9 minimal system
H = inverse(T2) @ reshape3x3(h) @ T1
check_rank_and_projection_denominators(H)
H = refine_geometric_error(H, pairs)

Setting the bottom-right coefficient to one is safe only when that coefficient is not near zero. Its value depends on coordinates, not on a universal camera-angle threshold. Avoid normal equations when QR or SVD can solve the fit with better conditioning.

Four pairs determine a homography only in a nondegenerate configuration, with no three points collinear in either image. More well-distributed correspondences improve estimation, but a cluster in one corner can still extrapolate poorly. DLT minimizes algebraic error; refine using projected pixel residuals for a geometric fit.

Why four, and when fewer will do. The number four is not magic; it is just the eight degrees of freedom of a projective homography divided by the two equations each correspondence supplies. Assume less motion and you need fewer pairs, because every assumption you are willing to make removes degrees of freedom. A full projective map has 8 DOF → 4 pairs. Forbid perspective convergence (keep parallel lines parallel) and you drop to an affine map, 6 DOF → 3 pairs. Allow only translation, rotation, and a single scale (a similarity) and it is 4 DOF → 2 pairs. Adding translation to in-plane rotation and scale gives those 4 DOF; rigid translation-plus-rotation is 3 DOF → still 2 pairs (one is not quite enough); and pure translation is 2 DOF → one pair. The lesson recurs all through the book: a stronger model is cheaper to fit and more robust to noise, but only pays off when the assumption is actually true. Use the simplest transform the geometry allows, and no simpler. (For panoramas specifically, a known focal length turns the inter-image map into a 3-DOF rotation, fittable from as few as two correspondences. That is the basis of the rotation-only stitchers in the next chapter.)

20.1.6 Warping and assembling the panorama🔗⧉

With $H$ in hand for each frame, we build the wide image. Pick one image's coordinate frame as the canvas, the reference, and for every other image, warp it into that frame using its homography (warping each frame one by one toward the reference). The union of the warped frames is the panorama; output pixels that no source image reaches remain invalid or transparent.

Inverse warping evaluates each covered output location in its source image. It avoids forward-splat sampling gaps, but cannot fill areas outside all source footprints. Carry a validity mask instead of using black as missing data. A corner bounding box is valid only when no projective pole crosses the source rectangle; check denominator signs and impose finite canvas limits before allocation. Resample with antialiasing appropriate to local minification (Chapter 11.1, Warping).

Two loose ends, both pointers. First, the warped frames overlap and will not match exactly (exposure, vignetting, and tiny misalignments differ between shots), so they must be merged seamlessly; that is the job of the blending section (feathered, two-scale, or Poisson). Second, warping everything onto a single flat reference plane stretches badly at very wide fields of view; cylindrical or spherical projections (and the stereographic "little planet") are the fix, covered later under other projections.

20.1.7 Another application: document flattening and merging🔗⧉

Everything above needed pure rotation because general scenes have depth. But there is a second route to the same homography, and it relaxes the rule entirely: a planar scene. If the scene is a single plane, a homography relates any two views of it, even when the camera translates, because a plane imposes one depth relationship across its whole surface, with no independent per-pixel depth to cause parallax. A plane photographed from any viewpoint maps to a plane by a homography. This is the planar twin of the rotation case.

Document rectification maps four detected page corners to a rectangle. Set its aspect ratio from known page dimensions or another justified constraint; arbitrary target dimensions can stretch the page. A homography cannot flatten a curved book page or remove lens distortion by itself (Figure 20.1.6⛶ fullscreen).

The same machinery also handles document merging. To capture a document or whiteboard too big for one frame, shoot overlapping planar patches and stitch them with planar homographies, the identical pipeline as a panorama, except the homography is justified by the flat scene rather than by pure rotation. This is the basis of book scanning, mosaicing microscope slides, and tiling aerial or satellite images of locally flat ground.

Panoramas (rotation) and document rectification or merging (plane) are the same homography solve; only the geometric reason the homography is valid differs. Perspective-distortion correction (straightening a building's converging verticals) is the very same trick, and is developed alongside the warp machinery in Matching pixels across space and time.