30.7 Time-lapse photography🔗⧉

A time-lapse samples the world slowly. Instead of sampling the world thirty times a second, it takes one frame every few seconds, minutes, or hours, and then plays those frames back at ordinary video rate, so a flower's day-long unfurling, a glacier's decade-long retreat, or a city's afternoon of traffic compresses into a few seconds. It is the exact temporal mirror of slow-motion. Slow motion plays captured or synthesized frames more slowly relative to scene time (Chapter 30.4, Frame interpolation and slow-motion synthesis). A time lapse captures widely spaced instants, or selects them from denser footage, then plays them closer together. Both are operations on the time axis; they just point in opposite directions.

What makes computational time-lapse interesting and hard is not the capture, but the coherence. When successive frames are 1/30 of a second apart they look almost identical, which is the whole premise of video compression (Chapter 30.2, Video compression and motion compensation). When they are an hour apart they look wildly different: the sun has swung across the sky, the white balance has drifted from warm morning to blue noon, the auto-exposure has chased a passing cloud, and a long shoot may have nudged the tripod. The frames carry a slow, intended change (the thing you came to film) buried under a fast, unwanted jitter (exposure, color, camera tremor). Every technique in this chapter is, at heart, a way to keep the intended change and suppress the jitter across that enormous baseline.

30.7.1 Capturing a time-lapse: interval, shutter, and day-to-night transitions🔗⧉

The capture instrument is the intervalometer: a timer that trips the shutter once every $T_{\text{interval}}$ seconds. Play the resulting frames back at $f_{\text{play}}$ frames per second and you get a speed-up factor

since each real interval of $T_{\text{interval}}$ seconds is replayed in $1/f_{\text{play}}$ seconds. A frame every 30 seconds, played at 30 fps, runs the world at $30\times 30 = 900\times$, a four-hour sunset in sixteen seconds. The choice of $T_{\text{interval}}$ is the choice of how much time you remove, and it interacts with motion in exactly the way the sampling story predicts.

Concretely, the failure mode is strobing. A crisp, short shutter at a long interval makes moving content teleport: clouds jump in discrete leaps, pedestrians stutter across the frame, headlights become a dotted line rather than a ribbon. Each frame is a sharp snapshot of a position the eye never gets to connect to the next. The cure is to lengthen the shutter so each frame blurs over a meaningful chunk of the motion; by daylight that means stopping the lens down and adding a strong neutral-density filter so the exposure can run for a second or more without clipping. The streaked clouds and silky light-trails of a good time-lapse are not an accident; they are the photographer deliberately prefiltering the time axis before sampling it (Figure 30.7.1⛶ fullscreen).

The hardest capture problem, which time-lapse photographers call the holy grail, is the seamless day-to-night transition. A scene that runs from bright afternoon into night spans something like fifteen to twenty stops of brightness, far more than a single fixed exposure can hold without either clipping the sky at the start or crushing the streetlights at the end. So the exposure must be ramped across the shoot, opening up gradually as the light fails, and the changing aperture, shutter, and ISO then have to be reconciled in post so the playback brightens and darkens smoothly rather than lurching at every settings change. It is a moving-target exposure problem, a close relative of the multi-exposure merging in HDR merging spread out over thousands of frames instead of a single bracket.

30.7.2 Deflickering: stabilizing exposure, white balance, and light🔗⧉

Even when you intend a fixed exposure, a long sequence flickers. Several causes combine: mechanical aperture blades never stop at exactly the same diameter twice, auto-exposure and auto-white-balance hunt frame to frame, drifting clouds and changing daylight modulate the illumination, and artificial lights pulse at the mains frequency. The result is a high-frequency brightness-and-color jitter riding on top of the slow change you actually want. It is the photometric twin of the camera-path tremor that stabilization removes, only now it lives in the intensity of the pixels rather than their position (Figure 30.7.2⛶ fullscreen).

Deflickering removes it with the same philosophy as trajectory smoothing: estimate the unwanted per-frame perturbation and subtract it, while preserving the slow intended trend. The workhorse model is a per-frame affine photometric correction,

a gain $a_t$ and an offset $b_t$ (and, for color flicker, a per-channel white-balance scale) chosen so that static regions agree from frame to frame. The constraint is exactly the registration constraint of HDR and panoramas, transposed to the time axis: a static patch under unchanged illumination should agree across frames, but a static surface can also change legitimately as the light changes. Estimate $(a_t,b_t)$ against a chosen smooth photometric trend, rather than declaring every discrepancy to be flicker. Crucially you do not flatten the sequence to a single constant; that would erase the sunset you came to shoot. Instead you low-pass the corrected luminance over time: keep the slow, deliberate ramp, strip the frame-to-frame jitter. Chained across a long sequence the corrections can themselves drift, so they are typically tied back to a smoothed reference trace rather than to the immediately preceding frame, the same anti-drift discipline as the cumulative-path smoothing in Chapter 30.6, Video stabilization and rolling-shutter correction.

30.7.3 Hyperlapse: stabilizing a time-lapse that also moves🔗⧉

So far the camera has stood still. A hyperlapse is a time-lapse whose camera travels, a helmet- or dash-cam recording a long hike, bike ride, or drive, then sped up perhaps ten- or twenty-fold. Speed-up is merciless to a moving camera: every small wobble of the original walk, harmless at normal speed, becomes a violent jolt once ten seconds of footage are crushed into one. Naively dropping nine of every ten frames and applying ordinary 2-D stabilization can fail: the frame-to-frame motion is now so large and so erratic that the crop window cannot possibly keep up, and the result is a large lurch (Figure 30.7.3⛶ fullscreen).

Kopf, Cohen and Szeliski (2014), in First-Person Hyperlapse Videos, solve it by going fully 3-D, which is exactly the estimate→smooth→re-render spine of Chapter 30.6, Video stabilization and rolling-shutter correction, with a genuinely translating camera instead of a mostly-rotating one. They first reconstruct the camera's actual path and a coarse 3-D proxy of the scene by structure-from-motion, then fit a smooth target path through space, the trajectory a steadicam-on-rails would have followed, and finally render each output frame along that smooth path by stitching and proxy-warping pixels from whichever input frames saw that part of the scene. Because near and far content parallaxes differently under real translation, no single 2-D crop could ever produce this; only a 3-D-aware re-render can. Joshi and colleagues (2015), in Real-Time Hyperlapse Creation, trade some of that quality for speed with a clever approximation that runs on a phone: rather than reconstruct the scene, they choose a sparse, well-spaced subset of frames whose viewpoints already form a smooth-ish path (so the speed-up and the smoothing are decided jointly) and then apply lightweight 2-D stabilization to the result.

30.7.4 Mining time-lapses from the internet🔗⧉

You do not have to shoot a time-lapse at all. For any sufficiently famous subject, one has already been shot, by thousands of strangers, over years. Every popular landmark, glacier, or skyline sits in millions of public photographs taken from countless positions across all seasons and times of day. Buried in that unstructured pile is a time-lapse of the place's slow transformation, waiting to be assembled.

Martin-Brualla, Gallup and Seitz (2015), in Time-Lapse Mining from Internet Photos, extract exactly that. They take a huge community photo collection of one scene and align every image to a common virtual camera using the structure-from-motion machinery of Chapter 28.5, Photo tourism, recovering where each photographer stood and reprojecting all of them onto one shared viewpoint. They then sort the images by their capture date, turning a pile into a sequence, and, because those photos span every conceivable lighting and weather, they normalize the illumination so that the only thing left changing is the slow story they care about: a glacier retreating across a decade, a building rising floor by floor, a waterfall's seasonal swell. The temporal-coherence problem here is the same one as in a shot sequence: suppress the fast nuisance variation, keep the slow intended change, but stretched to its extreme, with a different camera, a different day, and a different photographer behind every single frame. The same long-baseline-change instinct, expressed through explicit 3-D viewpoints rather than a flat reprojection, drives Chapter 36.9, Re-photography of standing where a historical photographer stood and capturing the same scene again decades later (Figure 30.7.4⛶ fullscreen).

The project page collects many of these mined sequences (glaciers, skylines, waterfalls) as videos worth watching in motion: people.csail.mit.edu/mrub/timelapse.

30.7.5 Factoring a time-lapse: reflectance, illumination, and intrinsic images🔗⧉

A time-lapse of a static scene over a day provides information a single photo cannot. As the sun arcs overhead, the reflectance of every surface (its intrinsic albedo, the paint and stone and grass) stays fixed, while the shading (the sunlight, sky light, and the shadows they cast) sweeps continuously across the frame. That clean separation, what stays constant versus what moves, is precisely the wedge that makes the otherwise-ill-posed intrinsic-image decomposition tractable. Write each frame as the product

with reflectance $R(x)$ independent of time and a shading term $S_t(x)$ that varies frame to frame. Over a single still image you could never tease the two apart: a dark pixel might be dark paint in bright light or bright paint in shadow, but a sequence provides repeated evidence about fixed reflectance. Time-independent shading remains ambiguous with reflectance; the sequence alone does not identify the factorization (Figure 30.7.5⛶ fullscreen).

Weiss (2001), in Deriving Intrinsic Images from Image Sequences, makes this exact observation and uses it: working in the log domain, where the product becomes a sum, he estimates reflectance filter responses from temporal medians of log-image filter responses, then reconstructs the reflectance image. This relies on sparse, approximately zero-centered illumination responses; a permanent shadow can remain in the estimate. The intuition is the robust-statistics one from denoising: a shadow edge sweeps past any given pixel only briefly, so at each location the genuine reflectance gradient is the value that persists across the sequence while the transient shading gradients are outliers the median rejects. Sunkavalli, Matusik, Pfister and Rusinkiewicz (2007), in their factored time-lapse video work, push further by modeling the illumination physically as a sun component plus a sky component, factoring a day's footage into reflectance and these illumination layers so that one can relight the scene, remove the moving shadows, or recolor the sky after the fact. Both belong properly to Chapter 24.4, Intrinsic images with time lapse, which develops the decomposition in full; here the narrower point is this: a time-lapse observes a natural lighting experiment, and the very thing that made capture hard (the light won't hold still) is the thing that makes this decomposition possible.