37.3 3D displays with accommodation🔗⧉

The fixed-focus stereo displays discussed so far can present scene-dependent vergence while leaving accommodation tied to the display's optical plane. Recall the two cues from Chapter 5.8, Photographic representation and its limits. Vergence is the inward rotation of the two eyes to fixate a point: a near point makes them cross sharply, a far point makes their gaze lines nearly parallel. Accommodation is the focusing of each eye's lens: ciliary-muscle contraction reduces zonular tension so the lens becomes rounder for near focus; relaxation increases tension and flattens it for far focus. In the real world these two are yoked: they always report the same distance, and the visual system has wired them into a reflex (focus on something and your eyes automatically converge to it, and vice versa). A stereo display breaks the yoke.

Here is exactly how it breaks. A headset or 3-D screen produces a virtual object, say a butterfly that appears to hover 40 cm in front of your nose. To make it appear there, the display feeds your two eyes the disparity that a real object at 40 cm would produce, so your eyes converge to 40 cm: the vergence cue says "40 cm." But the light actually leaves a screen (or a virtual image plane) at some fixed optical distance, maybe 2 m, so to see the butterfly sharply each eye must accommodate to 2 m: the focus cue says "2 m." The two cues disagree, the yoke is fighting itself, and that is the vergence–accommodation conflict. It causes eye strain, headaches, and fatigue; it makes virtual objects look subtly wrong; and because the disagreement grows as objects come closer (vergence changes fast at near distances while the screen stays put), it is worst exactly where you most want to put virtual content, at arm's length, where you would read text or manipulate an object.

37.3.1 Four ways to deliver a focus cue🔗⧉

Varifocal. The simplest idea: keep the one image plane a headset already has, but move it to the depth the viewer is looking at. An eye tracker measures where the eyes are converging (their gaze fixation depth), and an actuator (a mechanically translated display, a tunable-focus lens, or a deformable mirror) slides the focal plane to match. Now the plane the eye focuses on coincides with the depth the eyes converge to, so vergence and accommodation agree at the point of regard. Varifocal is appealing because it spends almost no extra display resolution; it just needs good, fast eye tracking and a way to shift focus. Its limitation is the rest of the scene: only the gazed-at depth is correct, and everything else is rendered with synthetic blur to imitate the eye's real depth of field. Several research prototypes take this route.

Multifocal. Rather than moving one plane, present a few fixed focal planes at once, say three or four, stacked at different optical distances, and distribute the scene's content among them by depth, often blending across the nearest two ("depth-filtered" rendering). The eye can then accommodate to whichever plane holds the content it is looking at, no eye tracking required. Akeley and colleagues built an influential bench prototype in 2004 Akeley et al. 2004, a fixed-viewpoint display with three real focal distances created by beam-splitters, that demonstrated subjects accommodating correctly to the right plane. The trade-off is that a handful of discrete planes only approximates a continuous range of depths, and stacking them costs light, hardware, or time-multiplexing.

Near-eye light field. Put a microlens array in front of the microdisplay and you build, inside the headset, a tiny Chapter 31.2, Light field cameras: the display+lenslet sandwich reconstructs a light field, a bundle of rays with the correct directions, rather than a flat image. Because the eye then receives rays that genuinely diverge as if from points at various depths, it can focus anywhere within the supported range, and the focus cue comes out right. Lanman and Luebke demonstrated exactly this in 2013 Lanman & Luebke 2013 with a thin near-eye light-field display. This is the cleanest expression of the part's spine, that a display is a light-field emitter, but it pays the spatial–angular tradeoff of Chapter 31.2, Light field cameras in reverse: every ray direction you emit is spent out of the display's fixed pixel budget, so reconstructing a focusable light field sharply costs spatial resolution, and the result tends to look soft. (It also has a finite display depth of field, the dual of the camera's, with the same prefiltering issue treated for Chapter 37.4, Lenticular displays.)

Holographic and retinal. The two most exotic options. A holographic near-eye display uses a phase spatial light modulator (Optical modulators (spatial light modulators): DMD and LCD/LCoS) to synthesize the actual optical wavefront the scene would have produced, so the light's curvature, and therefore the focus cue, is correct by construction. Maimone, Georgiou, and Kollin demonstrated compact holographic near-eye displays in 2017 Maimone, Georgiou & Kollin 2017; the full story is Chapter 37.5, Holography. A different tack, retinal (Maxwellian-view) projection, scans the image directly onto the retina through a tiny exit pupil; with a small enough pupil the image is in focus regardless of the eye's accommodation (everything is sharp, like a pinhole), which weakens the display's focus cue rather than supplying the correct one, a different bargain with its own limits (a tiny eyebox, no real blur).

37.3.2 The other gaps, and the ultimate display🔗⧉

Accommodation is the headline problem, but a display "indistinguishable from reality" must close several gaps at once, and they trade against one another. Field of view: human vision spans roughly 200° horizontally; headsets cover maybe half that, and widening it strains optics and pixel budgets. Angular resolution: to match the eye's acuity (about one arc-minute) across a wide field would require roughly 12,000 samples horizontally across 200°, before accounting for optics and the desired sampling criterion. A full two-dimensional field would require many millions of samples per eye. The standard escape is foveated rendering: the eye only resolves fine detail in the central fovea, so track the gaze and spend full resolution there while rendering the periphery coarsely, a perceptual budget that reduces rendering work. Foveated rendering alone does not increase physical panel resolution; gaze-contingent optics or displays are needed to concentrate display samples themselves. Form factor: the dream is ordinary eyeglasses, not a ski-goggle brick, which is what drives pancake and waveguide optics. And in see-through augmented reality there is a hard problem the opaque headsets dodge: mutual occlusion: a virtual object should be able to hide the real world behind it, but additive see-through optics can only add light, never subtract it, so virtual objects look like translucent ghosts; properly occluding the real background needs a per-pixel light blocker the field is still working out.

The target that organizes all of this is Ivan Sutherland's 1965 essay The Ultimate Display Sutherland 1965, which imagined a display so complete that "a chair displayed in such a room would be good enough to sit in," a synthetic world the senses cannot distinguish from the real one. Three years later he built Chapter 37.2, VR goggles, toward that goal. The chapters of this part are the rungs between that vision and a flat screen, and accommodation is the rung where the easy tricks run out and the optical distribution reaching the pupil must carry appropriate focus information, whether through focal planes, rays, or a reconstructed wavefront, which is precisely where Chapter 37.5, Holography begins.