37.0 INTEGRAL AND IMMERSIVE IMAGING⧉
A photograph is flat, and Chapter 5.8, Photographic representation and its limits, laid out the three things a flat picture can do about the depth it threw away: ignore the loss, fake the missing cues with pictorial tricks, or actually break the limitation and put the depth back. This part covers display and capture methods that restore depth cues. Everything here is a display or capture technology that tries to return to the eye the depth, parallax, and presence the flat medium took away, from sending a different image to each eye to tracking the head and re-rendering, glasses-free lenslet screens, and reproducing the light's own wavefront.
The world is full of cues to depth, and a flat picture delivers pictorial cues but does not reproduce the scene's full binocular and focus cues. Chapter 5.8, Photographic representation and its limits, cataloged the cues the visual system uses: binocular disparity (the two eyes see slightly different images), motion parallax (near things slide against far ones as you move), vergence (the eyes rotate inward to fixate a near point), and accommodation (each eye's lens focuses to a particular distance). An ordinary photograph or monitor gives pictorial cues such as perspective, occlusion, shading, and familiar size. Binocular disparity and focus instead specify the physical display surface. The technologies in this part are best understood as a ladder, each rung adding back one cue the rung below could not. Conflating the rungs is the field's most common confusion, so we keep them strictly apart from the start.
The rungs, in order:
- Binocular disparity: a different image to each eye. This is the oldest fix, older than photography itself, and it is the additional scene-depth cue that stereo glasses and 3-D cinema primarily deliver, along with the vergence it drives. A fixed stereo photo adds disparity to its pictorial cues: turn or move your head and the scene is painted on, frozen.
- Head-motion parallax: the rendered view updates as the head moves, so near objects slide against far ones and the scene feels solid and anchored in the world. This is a genuinely different cue from disparity (it works with one eye shut), and it is what a tracked VR headset adds that a stereo film cannot.
- Room-scale walking: full 6-degree-of-freedom locomotion through the space: you physically move your body and explore the scene from any vantage. Another step beyond head-motion parallax, and one that demands view synthesis over a volume of viewer positions, from suitable surface geometry, ray samples, or a volumetric representation. A single panorama cannot support that translation.
- Accommodation: the eye's lens actually focuses at the right distance, in agreement with where the eyes converge. Almost nothing gets this right. Stereo glasses and conventional fixed-focus headsets show the picture on a screen at one fixed optical distance, so the eyes converge to the virtual depth but focus on the screen. This is the vergence–accommodation conflict that fatigues viewers and recurs from Chapter 37.1, Stereo glasses to Chapter 37.3, 3D displays with accommodation. Varifocal, multifocal, light-field, and holographic designs address that gap with different constraints.
The organizing idea of this part is integral imaging, and it connects directly to the capture half of the book. In 1908 Gabriel Lippmann proposed integral photography: photograph a scene through a sheet of tiny lenslets (a fly's-eye array) onto one plate, so each lenslet records its own slightly shifted view, and then replay the plate through the same array to re-emit the rays in their original directions. The forward direction is the ancestor of the plenoptic camera of Chapter 31.2, Light field cameras; the backward direction is the ancestor of the autostereoscopic display of Chapter 37.4, Lenticular displays. Capture and display are duals. A 3-D display is a Chapter 31.1, Light fields 101 emitter: it reproduces the rays a plenoptic camera would record, and almost every result about light-field cameras (the Chapter 31.4, Refocusing and synthetic aperture, Chapter 31.6, Light field aliasing and 4D Fourier analysis) has an exact mirror image on the display side. Hold onto this; it makes the whole part one idea.
The chapter sequence is as follows. Chapter 37.1, Stereo glasses, is the first rung and the historical root: Wheatstone's 1838 stereoscope, the Victorian parlor craze, and the channel-separation methods (anaglyph, polarization, shutter, wavelength) that route a different image to each eye in today's 3-D cinema. Chapter 37.2, VR goggles, adds the second and third rungs: a head-mounted display that tracks the head and re-renders, and the strict separation of stereo, head-motion parallax, and walking-through. Chapter 37.3, 3D displays with accommodation, is the frontier: the vergence–accommodation conflict and the varifocal, multifocal, near-eye-light-field, and holographic displays that try to fix it and reach Sutherland's "ultimate display." Chapter 37.4, Lenticular displays, is glasses-free 3-D: lenslet sheets and parallax barriers, Lippmann's integral photography as the origin, and the resolution-versus-views tradeoff that is the plenoptic capture trade run in reverse. And Chapter 37.5, Holography, is the limit case: the only display that reproduces the optical wavefront itself, so a sufficiently accurate reconstruction can supply accommodation and other depth cues, subject to aperture, bandwidth, viewing-zone, and tracking limits, from Gabor's 1948 invention to today's neural computer-generated holograms.
Contents of this part
- 37.1 Stereo glasses
- 37.2 VR goggles
- 37.3 3D displays with accommodation
- 37.4 Lenticular displays
- 37.5 Holography
- 37.6 Retinal projection