17.4 Ghosts and motion⧉
Take three exposures a second apart, merge them, and look at the person in the frame. They will be in three places at once, faintly, each at the weight its exposure carried. That is a ghost, and it is what separates a demonstration of high dynamic range imaging from a product.
Separate global camera motion, independently moving subjects and changes in visibility. A global transform can align a background; local correspondence can also align moving subjects where matching content remains visible. Occlusion, blur and clipping leave regions with no reliable correspondence.
17.4.1 Align first, and know what alignment cannot do⧉
A handheld bracket moves. Even a steady hand shifts the camera by several pixels between exposures a fraction of a second apart, and a merge of misregistered frames doubles every edge, at the weights each exposure receives. Before any merge, the frames have to be brought into register, and this is harder than ordinary image alignment because the frames do not look alike: a dark region that is black in the short exposure is mid-gray in the long one, and a window that is detailed in the short one is a white blob in the long one.
What can be compared. There are three ways around the exposure difference. With linear raw frames, dividing each by its exposure factor $k_i$ puts them on a common radiance scale (Chapter 17.3, Merging exposures), and we can compare them directly using only pixels that are well exposed in both, neither clipped nor lost in noise. With processed frames whose response is unknown, we compare a representation that does not depend on exposure: the ranks of the pixel values, as in Ward's method below, or local structure such as gradients, edges and keypoints, which survive a change of brightness as long as they are not clipped. Correlation measures normalized by the local mean and variance tolerate a change of gain and offset, but not clipping.
Ward's median-threshold bitmap. Ward's method Ward 2003 (MTB) rests on one observation. Each exposure maps radiance to pixel values through an increasing function, a different one per frame but increasing in every frame. An increasing function does not change the order of the pixels, so the half of the pixels above the median in one frame is the same half of the scene in every frame. Thresholding each frame at its own median therefore gives nearly the same bitmap whatever the exposure (Figure 17.4.2). The bitmaps disagree only where the ordering is lost: pixels clipped together at white or at black, and pixels close to the median, where noise decides which side they fall on. The method excludes the second kind explicitly, with an exclusion bitmap that ignores every pixel within a small tolerance of the median.
Aligning two bitmaps is cheap. For a candidate shift, we take the exclusive or of one bitmap with the shifted other, which marks the pixels where they disagree, mask out the excluded pixels, and count the remaining ones; the best shift has the fewest disagreements. These are bit operations, many pixels per machine word. To find shifts of tens of pixels without testing every one, the search runs on an image pyramid. At the coarsest level it tests the nine shifts of $-1$, $0$ or $+1$ pixel in each direction and keeps the best; at each finer level it doubles that shift and again tests the nine shifts around it. With $L$ levels this finds shifts up to $2^L-1$ pixels in each direction while testing only nine candidates per level.
mtb_align(ref, img, levels, tol): # returns the integer shift of img onto ref shift = (0, 0) for level = levels-1 down to 0: # coarsest first a = gray(downsample(ref, 2^level)) b = gray(downsample(img, 2^level)) Ba = a > median(a); Ea = |a - median(a)| > tol # bitmap and exclusion map Bb = b > median(b); Eb = |b - median(b)| > tol shift = 2 * shift best = argmin over (dx, dy) in {-1, 0, 1}^2 of count( (Ba xor move(Bb, shift + (dx, dy))) and Ea and move(Eb, shift + (dx, dy)) ) shift = shift + best return shift
The median is the natural threshold because it splits every frame into equal halves. It fails when it falls where one of the frames has lost the ordering: if more than half of the long exposure is clipped, its median is white and its bitmap carries no information. A threshold at another percentile, chosen so that it lies in the range both frames record, avoids this. For the same reason, it is better to align each frame to its neighbor in the bracket, whose exposure overlaps most with it, and compose the shifts, than to align the shortest frame directly to the longest.
mtb_align, one level per search step, coarsest first, on $L$ levels of 2 × 2 box averages. At each level both frames are thresholded again, the start is twice the previous shift, and each of the nine candidates around it scores the number of pixels where the reference bitmap and the moved long bitmap disagree, counting only pixels kept by both exclusion bitmaps; the fewest wins. The panels show the two bitmaps at that level and, in red, the disagreements of the candidate outlined in the grid. At the default the shift grows through (−1, 0), (−2, +1) and (−3, +2) to (−7, +4), which undoes the shake exactly. A shake beyond $2^L-1$ pixels is out of reach: with $L=4$, a shake of 20 pixels stops at 15, and $L=5$ finds it. With "design center" at a 7.5-stop gap the median search lands on (−9, −1); a threshold at the 10th percentile, which lies where both frames still record detail, finds (−7, +4) again. Radiance maps: Byong Mok Oh. Interactive: move the search step to walk down the pyramid, click a candidate to see its disagreements, and change $L$, the tolerance, the percentile or the shake.Beyond translation. The bitmap search finds an integer translation. A handheld camera also rotates, slightly about the optical axis and more about the other two, which for a distant scene moves the image as a homography. Keypoint matching on the exposures, with a robust fit of a homography to the matches, handles that case Tomaszewska & Mantiuk 2007, and the bitmap or correlation alignment can be refined to sub-pixel precision; an error of half a pixel is invisible in one frame but softens fine detail in a merge.
After global alignment, what remains cannot be fixed by a single transform. Camera translation through a scene with depth creates parallax: near and far objects shift by different amounts, and no one transform aligns both. A rolling shutter records rows at different times, so a fast pan shears the frame. Subjects that moved on their own are displaced, and local flow, discussed below, can correct some of that displacement, but it cannot expose a hidden background or restore detail blurred within an exposure. Those regions are the subject of the rest of this chapter.
17.4.2 Rejection: pick a reference and disagree with it⧉
A common design chooses a reference and reduces contributions from frames that disagree with it after registration and exposure normalization.
Test disagreement relative to expected noise. For independent radiance estimates $q_i$ and $q_r$ with variances $v_i$ and $v_r$, a useful diagnostic is
Here $\epsilon$ is a small variance floor in the same units. A large residual suggests an inconsistency, not specifically motion: registration error, response mismatch and changing light can also cause it. Correlated interpolation errors require a covariance correction. Reject or reduce suspect contributions, retaining a flagged reference fallback where necessary.
Precision weighting can reduce a ghost when the disagreeing frame is also noisy. It cannot reject a well-exposed moving object merely because it moved: two confident observations can describe different scene states. Judge noise weighting and motion rejection separately on scenes where the subject is valid in several exposures.
Three consequences follow, and they explain why the results look the way they do.
The reference decides the composition. Whatever moved is rendered as it was in the reference frame, so the choice of reference is an editorial choice: the middle exposure is the usual default because it has the most well-exposed pixels, and phone pipelines choose the sharpest frame instead (Chapter 17.6, HDR+ and burst imaging on phones).
Rejection costs range exactly where it was needed. A moving subject against a bright window is the canonical hard case: the frames that would have recovered the window are the ones being rejected around the subject's edge, so the result is a correctly exposed window with a halo of noise or clipping around the person.
Rejection makes the effective sample count and output uncertainty spatially varying. Pass that information to later processing; a global noise level cannot describe both a single-frame fallback and a well-supported average.
Imagine a person crossing a doorway between two bracketed frames. After exposure normalization, both observations at the person's boundary can be well exposed and have low predicted sensor variance, yet represent different surfaces. Inverse-variance weighting will confidently average them and create a ghost. A consistency test must compare their disagreement with the noise expected from the measurements and account for registration uncertainty. If it rejects the nonreference observation, that location may retain only one frame and therefore more noise. Deghosting trades available averaging evidence for a coherent choice of scene content; noise weighting alone does not identify motion.
Reliability weighting and motion rejection answer different questions. A badly exposed observation may already receive little weight, so its disagreement contributes little to the merge even without a motion test. But imagine a dark sleeve moving over a wall when both sleeve and wall are comfortably within the usable range of several exposures. Their values can each be precise measurements of different scene content. Inverse variance alone has no reason to reject either. A correspondence or consistency test must decide which observations belong together. Checking this case prevents a favorable example of noise weighting from being mistaken for a general deghosting solution.
17.4.3 The other approaches, and why rejection is the usual choice⧉
Two other families exist and are worth knowing.
Optical-flow methods warp local scene motion before merging. Good correspondence can preserve exposure information on moving subjects. Incorrect flow, occlusions and differing motion blur can produce distorted edges, so confidence and fallback remain necessary.
Patch-based reconstruction searches other exposures for content compatible with a chosen reference (Sen et al. 2012 (patch-based HDR)). It can transfer measured detail where appropriate matches exist and infer missing content elsewhere. Those cases have different evidential value; a patch search is not automatically pure invention.
Learned merges now do all of this in one network, taking the aligned stack and producing the result, and they inherit the same tradeoff with the choice buried in the training data rather than in a threshold.
Choose the fallback according to the task. A noisy reference may be preferable to a doubled face, but clipping away a required measurement can also be unacceptable. Report the unsupported region rather than hiding the loss of data behind a clean-looking reconstruction.
The fallback is also a rendering preference. A patch of visible grain tells the viewer that the capture was difficult; a sharply doubled or warped face can misdescribe what happened. A photographic pipeline may therefore accept extra noise to preserve a coherent subject. That choice belongs in the design and evaluation, since a scalar reconstruction loss does not by itself express every consequence of the failure.
17.4.4 The bracket you did not have to take⧉
Shorter capture windows reduce changes between frames. Shared-integration sensor designs avoid bracket timing differences, while short bursts simplify exposure comparison. Both can still contain motion blur, occlusions, flicker and rolling-shutter distortion.
A fixed-exposure burst makes radiometric comparison simpler. Later capture designs can also use brackets when longer frames improve useful shadow information; the motion and noise tradeoff must be evaluated for that scene.