36.9 Re-photography⧉
A glacier photographed in 1900 and again today; a city street before and after a century of rebuilding; a coastline eroding decade by decade. Re-photography re-captures a scene from the same viewpoint as a historical photograph so that the two images, overlaid, reveal change over time. The genre is old, and the Rephotographic Survey Project is a canonical example, but doing it accurately by hand is difficult: a few centimeters or a degree off, and the overlay no longer registers. This chapter is about the computational version, which turns "stand in exactly the right spot" into a guided, automatic procedure.
36.9.1 Why you cannot just overlay⧉
The instinct is to align the new photo to the old one with a 2-D warp, the way panoramas are stitched. It does not work, for two compounding reasons. First, the scene has changed: buildings are gone, vegetation has grown, lighting and season differ, so large parts of the two images simply do not correspond, so the appearance-based alignment a stitcher relies on is corrupted. Second, and more fundamentally, a single homography can register a planar scene, or views related by pure camera rotation (the no-parallax case from panoramas). A historical photo was taken from some unknown 3-D position with an unknown camera, and unless you physically return the lens to that 3-D point, the parallax between near and far objects will not generally align under one global homography. Depth-dependent warps can change viewpoint, but require geometry and cannot reveal unobserved surfaces. You cannot warp your way to the right viewpoint; you have to go there.
So the right problem is relative-pose estimation: how is my current camera positioned and oriented relative to where the historical camera stood? With calibrated views this connects to the essential-matrix machinery of the opening chapter; historical cameras may require additional calibration and scene constraints: sparse feature matches between the two views, plus a RANSAC-based estimate of the relative pose $(R, t)$, applied across a punishingly long time baseline.
36.9.2 The real-time guidance loop⧉
Computational Re-Photography Bae et al. 2010 (Re-Photography) (Bae, Agarwala, and Durand) turns that estimate into a live guidance system, visual servoing for a human photographer (Figure 36.9.1). The loop runs continuously: from the current matched features, estimate the relative pose between the live view and the reference photo; from that pose, compute how the photographer should move to reduce it; display that as on-screen directions: "move left, rotate up, step back," plus the estimated translation vector; the photographer moves; re-estimate; repeat. As the live viewpoint converges on the historical one, the guidance arrows shrink to nothing, and the final frame can be captured and aligned for the overlay. The system handles the long-baseline matching robustly (the scene has changed, so most putative matches are wrong, and RANSAC must survive a high outlier rate) and guides the user smoothly despite a continuously shifting, noisy pose estimate.
36.9.3 Where it sits⧉
Re-photography is one of a family of long-baseline change tasks. The artistic "then-and-now" genre (Klett's survey project) is its cultural root; data-driven appearance hallucination Shih et al. 2019 addresses a related synthesis problem (matching the look, time of day and season, once the viewpoint matches); crowd-sourced viewpoint mining from photo collections (Part 28, Photo collections) and time-lapse mining (Chapter 30.7, Time-lapse photography) provide the temporal-change siblings; and Google Street View's "time travel" is the productized form, letting anyone scrub a registered viewpoint across years. The capture-and-alignment applications are documentary; synthesized appearance must be distinguished from observed change: architecture and urban change, glaciers and coastlines for climate science, ecology (vegetation and habitat over decades), and art history (re-occupying the viewpoint of a historical painting or photograph). What they all share is the chapter's core move: when you cannot align two images by warping, estimate the 3-D relative pose and physically close it, which is just the part's geometry, pointed at time.