31.6 Light field aliasing and 4D Fourier analysis⧉
The preceding discussion introduced a camera that refocuses after the shutter and an array that sees through a fence, and both rested on a quiet assumption we never examined: that you had captured enough views. Chapter 31.4, Refocusing and synthetic aperture shifts and adds a grid of sub-aperture images and trusted them to be dense enough that the sum came out smooth. Chapter 31.7, Lumigraph and shape priors for sharper light field rendering explains that they often are not dense enough (pure ray interpolation ghosts and blurs from sparse cameras) and patched it with a depth proxy, on the promise that "a little geometry buys back the samples you didn't take." This chapter makes both the warning and the cure exact. How many views is enough? When does too-sparse a grid produce a double image? And why does knowing depth let you get away with fewer cameras? These are not new questions. They are the sampling and aliasing questions of Chapter 6.3, Linearity, Fourier, Aliasing and deblurring, asked of a four-dimensional signal.
So we lean on the one idea from that chapter that organizes everything: a signal you sample on a regular grid has a spectrum that repeats, copying itself once per sampling period, and as long as the copies do not touch you can reconstruct the band-limited signal from its samples under the theorem’s assumptions. Let the copies overlap and high frequencies masquerade as low ones, which is aliasing, and no reconstruction can separate them. The goal is to keep the replicas apart, by sampling finely enough or by band-limiting the signal first so its spectrum is narrow enough to pack without collision. Lifting that game to the light field requires exactly one thing the 1-D story did not: knowing what the light field's spectrum looks like. That shape is remarkably constrained, and the constraint is the scene's depth.
31.6.1 The light field is a sampled signal, and its samples are viewpoints⧉
Recall the object. The light field $L(u,s)$ records the radiance of the ray that crosses the aperture (camera) plane at $u$ and the image plane at $s$ (the full field carries the second aperture axis $v$ and second image axis $t$; the slice $L(u,s)$ shows every effect). A real capture never has the continuous function. It has a grid of views, one image per camera position $u_0, u_1, u_2, \dots$ spaced $\Delta u$ apart, each image itself sampled in $s$ by its pixels. The light field is thus a signal sampled on a grid in all four dimensions, and the dimension that is usually the coarsest, and the one that matters most here, is the angular one: cameras are expensive, so $\Delta u$ is large. The question "did we capture enough views?" is, precisely, "is the viewpoint spacing $\Delta u$ small enough for the light field’s angular bandwidth?" To answer it we need the light field's spectrum.
31.6.2 Depth is slope is spectral orientation⧉
The principle is best seen in the epipolar-plane image (EPI) from Chapter 31.1, Light fields 101, the slice of the light field at one fixed scene row, with image position $s$ on one axis and camera position $u$ on the other (Figure 31.6.1). We met its central fact already: a single Lambertian scene point traces a straight line across the EPI, and the line's slope is the point's disparity, hence its depth. As you slide the camera by $\Delta u$, the point's image slides by $\Delta s$ proportional to $\Delta u$, with the constant of proportionality set by depth. A near point races across the frame (steep slope), a far point barely moves (shallow slope), a point at infinity does not move at all (the lines run vertical). A whole textured surface at a single constant depth $z$ fills the EPI with parallel lines of one shared slope $m(z)$.
Now take the 2-D Fourier transform of that EPI. A pattern that is constant along lines of slope $m$ in the $(u,s)$ plane has all of its energy on a single line through the origin in the frequency plane $(\Omega_u,\Omega_s)$, and that spectral line is tied to the depth, because $m$ is. Symbolically, a constant-depth textured plane gives $L(u,s)=h(s-m\,u)$ for some 1-D texture profile $h$, whose transform is
which reads: all the energy lies on the line $\Omega_u = -m\,\Omega_s$, with the texture's own spectrum $\hat h$ painted along it. One depth, one spectral line; the texture only decides how bright the line is along its length. Depth has become an orientation in the spectrum. This is the same fact the refocusing chapter used in the spatial domain (depth is a slope you read off the EPI), now read off the frequency domain instead.
31.6.3 The bowtie: a spectrum shaped by the scene's depth range⧉
A real scene is not one depth but a range of them, from a nearest surface at $z_{\min}$ to a farthest at $z_{\max}$. Each depth contributes its own spectral line through the origin, and as depth sweeps from near to far the line's orientation sweeps between two extremes. The energy of the whole scene therefore fills the wedge spanned by those two bounding orientations, and because $\Omega_s$ runs both positive and negative, the wedge appears as a symmetric double wedge through the origin, the shape everyone calls the bowtie (Figure 31.6.1, right). This is the central object of the chapter, and the result of Chai et al. 2000 (plenoptic sampling): under the idealized Lambertian, bounded-depth model, depth constrains the dominant spectral support to a bowtie. Finite windows, visibility boundaries, and non-Lambertian appearance broaden that support; exact band limitation is an approximation for real captures.
The wedge's half-angle carries the whole intuition. Disparity is proportional to inverse depth, so the two bounding slopes differ by an amount proportional to $\big(\tfrac{1}{z_{\min}}-\tfrac{1}{z_{\max}}\big)$. A flat scene, such as a painted backdrop at one depth, has $z_{\min}=z_{\max}$, the wedge collapses to a single line, and the light field is almost free to sample (every view is a shifted copy of one image). A scene with a near foreground against a far background has a large inverse-depth spread, a wide bowtie and a demanding sampling requirement. The bowtie's radial extent, on the other hand, is set not by depth but by texture: the finest spatial detail in the scene caps $|\Omega_s|$ at some band-limit $B$, so the bowtie is a wedge of finite reach, not an infinite cone. Depth sets the angle; texture sets the length.
This already explains a fact from Chapter 31.4, Refocusing and synthetic aperture that we stated without proof: refocusing is a shear. Shifting-and-adding by $k$ shears the EPI, which shears the bowtie in the spectrum, and integrating over the aperture (summing down the $u$ axis) keeps only the $\Omega_u=0$ line, a 2-D slice of the 4-D spectrum. That is exactly the Ng 2005 Fourier-slice statement: an aperture-integrated refocused image corresponds to a slice through the light-field spectrum, and choosing the focus is choosing the slice's orientation. We needed the spectrum's shape to make sampling rigorous; the same shape re-derives refocusing for free.
31.6.4 How densely must you sample? The plenoptic-sampling bound⧉
Sampling viewpoints at spacing $\Delta u$ replicates the spectrum at angular-frequency intervals $2\pi/\Delta u$. Let the spatial angular-frequency band limit be $|\Omega_s|\le B$, and let EPI slopes lie between $m_{\min}$ and $m_{\max}$. After centering the shear on their midpoint, the maximum residual angular frequency is $B(m_{\max}-m_{\min})/2$. A corresponding sufficient bound is $\Delta u<2\pi/[B(m_{\max}-m_{\min})]$, assuming adequate spatial sampling and the idealized support model. Since slope spread is proportional to inverse-depth spread for a fixed calibrated projection, this yields the scaling law:
Read it back in words: you may space your cameras no farther apart than the reciprocal of (the finest scene texture) times (the spread of disparities between the nearest and farthest object). This is the plenoptic sampling bound of Chai et al. 2000 (plenoptic sampling), and it says two intuitive things at once. Finer texture (larger $B$) demands denser views: a detailed scene has more to alias. And a deeper scene (larger inverse-depth spread) demands denser views: the nearer the foreground, the more violently it parallaxes between cameras, and the more views you need to track that motion without ghosting. Push the foreground to touch the lens and the requirement diverges; flatten the scene to a single plane and one view nearly suffices (Figure 31.6.2).
31.6.5 Geometry buys back samples: the depth-vs-views tradeoff⧉
The inverse-depth-spread bound already exploits the wedge’s orientation; it should not be described as the bound of a completely depth-blind axis-aligned filter. Reparameterizing about a reference depth makes simple reconstruction filters more efficient. Local geometry can reduce the residual disparity spread further: after warping images to an estimated surface, the renderer interpolates smaller remaining shifts instead of the full scene parallax. With exact geometry and Lambertian appearance, one view can supply radiance for points it actually sees. It cannot supply hidden surfaces, newly revealed texture, or unseen specular appearance.
This is the rigorous form of the promise Chapter 31.7, Lumigraph and shape priors for sharper light field rendering made. Its depth proxy was not a heuristic: it was a way to reparameterize the light field so the bowtie shears to axis-aligned, after which a cheap filter packs the replicas tightly and few cameras suffice. There is a genuine tradeoff curve here: views and geometry are substitutes. Capture many cameras and you need no scene model; supply a depth model and you need few cameras; any point on the curve in between is available. Isaksen et al. 2000 turned exactly this knob into an interactive control: their dynamically reparameterized light fields let the user steer the focal plane and aperture after capture, which in spectral terms is choosing where to place and how wide to open the reconstruction filter against the bowtie. Geometry and sampling density are the two prices of the same rendering quality, and you may pay in whichever currency is cheaper.
31.6.6 Under-sampling looks like a ghost⧉
What does the failure actually look like in a picture, when the cameras are too sparse and the replicas overlap? In the spatial domain, refocusing summed a grid of views, each shifted to align one depth. If the grid is too coarse, the views being summed are too far apart in viewpoint: between one camera and the next, a scene feature has jumped by more than its own width, and the sum lays down two (or more) offset copies of it instead of one smeared blur. The refocused image shows ghosting: doubled edges, a faint second image of a railing or a face, periodic replicas of a bright highlight (Figure 31.6.3). This is not noise and not ordinary blur; it is 4-D aliasing, the overlapping spectral replicas reappearing as structured phantom copies, the exact analog of the moiré and the backward-spinning wagon wheel of Chapter 6.3, Linearity, Fourier, Aliasing and deblurring. The replicas that should have stayed apart have folded a high angular frequency down onto a low one, and the eye reads the fold as a ghost.
Denser capture reduces angular spacing and separates spectral replicas. A genuine prefilter must act before undersampling; it reduces bandwidth at the cost of detail. After a sparse light field has already been recorded, reconstruction filtering can suppress some visible artifacts but cannot generally separate frequencies that have aliased together.
A larger synthetic aperture is not automatically an angular antialiasing filter for existing samples. Shift-and-add integrates along a chosen shear, and a sparse set of views can produce several displaced copies of an out-of-focus highlight. Increasing baseline at fixed view count can make those copies farther apart. More densely spaced views, appropriate reconstruction filters, or geometry-aware interpolation can make the defocus smoother. Distinguish the aperture extent, the number and spacing of views, and the prefilter used during capture.
31.6.7 Where this sits⧉
Step back and the chapter is a single sentence in a new dimension. A light field sampled on a grid of viewpoints is a sampled signal; its spectrum is a depth-shaped bowtie; sampling replicates that bowtie; overlapping replicas are ghosts; the cures are denser views or an angular prefilter, and geometry reshapes the bowtie so fewer views suffice. Every one of those clauses is a clause of Chapter 6.3, Linearity, Fourier, Aliasing and deblurring, and the only genuinely new physics is that the scene's depth range controls the spectrum's shape. It is also the theoretical floor under the whole part: it tells the camera-array designer how many cameras to buy, tells the Chapter 31.7, Lumigraph and shape priors for sharper light field rendering what its depth proxy is worth, and tells the Chapter 31.4, Refocusing and synthetic aperture when a sparse aperture will produce ghosted defocus. The same frequency analysis generalizes far beyond viewpoint sampling: Durand et al. 2005 (frequency analysis of light transport) carried it to the full transport of light through shading, shadows, and lenses, where every interaction is a predictable reshaping of a local spectrum, but that is a story for graphics. For photography, the lesson is the practical one the rendering chapters needed: how densely to sample, and how much a little geometry is worth.