20.5 Continuous panoramas (e.g. on cell phones)🔗⧉

A sweep panorama builds a wide view from a sequence captured while the camera turns. A streaming system estimates motion and updates a preview as frames arrive; it may also retain frames for a later refinement. Hand motion can include translation, but a sweep does not necessarily translate at every instant.

Strip mosaicking is one useful design: select narrow regions from successive frames and combine them into a panorama. It limits mixed viewpoints within each contribution, at the cost of discarding data. Actual products may use adaptive seams, wider regions or post-capture optimization; the strip model here is an explanatory baseline, not a verified description of every phone.

20.5.1 Incremental registration of a video stream🔗⧉

A streaming tracker can match each new frame to a previous frame or keyframe, predict motion from inertial measurements, and periodically check older overlaps. For an incremental homography chain, define $H_{t-1}$ from the previous frame to the mosaic and $\Delta H_t$ from the current frame to the previous frame:

With those directions, $H_t=H_{t-1}\Delta H_t$ maps the current frame into the mosaic. Gyroscope integration provides a rotation prediction after camera/IMU calibration and time synchronization. It does not measure translation, and gyro bias accumulates orientation error. Visual measurements refine the prediction.

Capture guidance can help keep consecutive frames overlapping and reduce motion blur. Stop or request recapture when the tracker loses support rather than appending a poorly registered strip. Smooth motion helps tracking but does not guarantee a well-conditioned estimate in a blank or repetitive scene.

Chaining transforms accumulates drift. Keyframe matching, loop closure and incremental or windowed optimization can reduce it; a later global refinement may also be practical. A gyroscope alone does not anchor absolute orientation, and a narrow strip does not guarantee that closure errors disappear.

20.5.2 Mosaicking a moving strip (and why a central strip)🔗⧉

In a strip-mosaic baseline, choose a narrow source region and warp it to the output surface (Peleg et al. 2000). Overlap adjacent contributions enough for registration and blending. For valid, aligned samples, a feather blend is

with $\alpha$ ramping across the seam exactly as in Chapter 20.3, Blending. The mosaic grows sideways as you pan, one strip at a time (Figure 20.5.1⛶ fullscreen).

The central region can reduce some lens-related distortions, but its advantages must be stated separately:

• Vignetting and radial distortion are often weaker near the principal point. A full-height vertical strip still extends far from it at the top and bottom, so it is not uniformly free of either effect.
• Narrow strips reduce the area over which mismatched viewpoints are mixed. They do not eliminate parallax: under lateral translation $b$, a point at depth $Z$ has disparity approximately $fb/Z$, even on the principal ray. Depth boundaries can still jump between strips.

Strip selection limits blending between different viewpoints. It can conceal some parallax but may create a multi-perspective image that has no single camera center. Inspect repeated structures and foreground boundaries before treating it as a central projection.

20.5.3 Rolling shutter and exposure drift🔗⧉

Two caveats that a tripod hides come to dominate the sweep, one geometric and one photometric.

A rolling-shutter sensor starts different rows at different times. A full-height vertical strip includes the full range of row times, however narrow it is horizontally. For zero-based row $y$, a simple timing model is

where $T$ is row exposure duration. Under a constant-angular-velocity approximation, camera-to-world orientation can be written

Here $[\omega]_\times$ is the skew matrix for body-frame angular velocity. This approximates a pose at each row's midpoint; it does not undo motion blur accumulated during $T$. Rotation-only rectification uses those poses and calibrated intrinsics. Translation requires depth or another correspondence model. Timestamp offsets, gyro bias and the true readout direction matter. Global-shutter CMOS sensors are a separate case (Part 30, Video).

Panning more slowly reduces pose change during both readout and exposure. Reducing strip width does not reduce row-readout time for a vertical strip. Row-wise correction and exposure-blur reduction solve different problems.

Exposure and white balance can vary through a sweep. Lock them when the scene range permits, or estimate consistent photometric corrections from valid overlap. Mask saturation and moving content, and retain genuine scene brightness variation. Feathering may soften residual steps but cannot guarantee an invisible ramp or undo clipped data.

A practical streaming loop should make failure explicit:

for frame in stream:
    calibrate_color_lens_and_timing(frame)
    prediction = integrate_calibrated_gyro(previous_time, frame.time)
    transform, confidence = track_against_keyframe(frame, prediction)
    if not sufficient_overlap_and_support(confidence):
        pause_mosaic_update_and_request_recapture()
        continue
    retain_frame_and_transform(frame, transform)
    update_preview_with_valid_samples(frame, transform)
check_loop_closure_and_refine_if_available()
render_original_frames_once_with_masks()

For capture, keep overlap, avoid fast pans and watch close foreground objects. For implementation, check closure, source validity and row timing before tuning the seam blend.

Follow a sweep from a dim room toward a window. Automatic exposure can decrease the scale of later frames while automatic white balance changes their channel ratios. An overlap then contains evidence about both geometry and a photometric mismatch. Estimating a gain or color correction from corresponding unsaturated samples can reduce the mismatch before inserting the next strip. Locking camera settings avoids some drift but may sacrifice usable range elsewhere in the sweep. Small residual differences can accumulate into a broad brightness or color ramp even when no individual seam is sharp, so inspect the entire panorama as well as neighboring strips.