BookNotesSlidesP-sets Syllabusdraft · v0.2.49.1
◀▲⌂▶
💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.

31.0 LIGHT FIELDS AND PLENOPTIC CAMERAS⧉

A camera throws something away at the moment of capture, and it does it on purpose. Each sensor pixel reports a single number, the integral of all the light rays that crossed the aperture and converged there, and in collapsing that cone of rays to one value it discards the one thing the rays still disagreed about: their direction. Which rays land on which pixel is exactly what the word focus names. This part is about what becomes possible if you refuse to sum them and record the rays themselves. Recall the splitting of the integral from Chapter 2.4, Image measurements as integrals over the plenoptic function: a pixel is an integral over the rays crossing the aperture, and once you keep the rays instead of collapsing them to the tally, you can re-split that integral any way you like afterward. This part exploits exactly that.

The object you get is the light field: the radiance along every ray, a function of where the ray is and which way it points. For a camera it is four-dimensional: two coordinates for the aperture and two for the focal plane, and from it any ordinary photograph is not a measurement but a computation you run afterward. That single shift, from capturing a picture to capturing the rays, is the recurring lesson L13.1, capture the full set, decide later, carried onto the aperture and angle axes, the siblings of the exposure axis (HDR), the viewpoint axis (panorama), and the focus axis (focal stacks). Record sufficiently dense spatial and angular samples and you can choose, after the shutter has closed, the focus, the aperture (and so the depth of field), the viewpoint, and even correct the lens's aberrations, all by re-sorting rays you already have.

Two things make the light field worth the trouble: one practical and one theoretical. As data, the light field is the true raw. A conventional raw file is "before the cooking" only up to demosaicking and tone; the light field is rawer still, the most complete record of what actually reached the lens, before the aperture even integrates it into a picture. It is, in that sense, a more complete acquisition record: within the captured aperture, viewpoints, bandwidth, and refocus range, photographs become projections selected afterward. A sampled radiance field still omits wave phase, often polarization, and most spectral information; “true raw” is an analogy for postponing angular integration, not a complete optical measurement. As a concept, the light field is the right variable for theory. For incoherent geometrical-optics models with fixed optics and linear sensing, pinhole, lens, coded-aperture, and array measurements can be written as linear projections of radiance (Chapter 6.7, Image formation as a linear system), ray space is the natural coordinate system in which to analyze cameras at all: it is what makes "how densely must I sample?", "how invertible is this system?", and "what does this camera actually measure?" tractable questions. The light field matters as much for what it lets you reason about as for what it lets you capture.

The part runs from the representation outward to the hardware and the applications.

Chapter 31.1, Light fields 101 builds the 4-D representation: capture-rays-not-pixels, the two-plane parameterization, the ray↔point duality, and the 2-D slices (pinhole images, angular views, and the epipolar-plane image whose line slopes are depth). Chapter 31.2, Light field cameras is how you actually record it: Lippmann's 1908 fly's-eye integral photography, the plenoptic / microlens camera (Adelson & Wang → Ng → Lytro), and the wide-baseline camera array (Wilburn), and it lands the spatial↔angular tradeoff that separates them. Chapter 31.4, Refocusing and synthetic aperture is what you do with the recorded field: refocusing as shift-and-add, a whole focal stack from one capture (the dual of Chapter 23.3, Focal stacks and depth of field extension), the Fourier-slice fast version, and the synthetic aperture that grows so wide it sees through occluders. Chapter 31.8, Light field microscopy is the plenoptic camera applied to the microscope: single-shot 3-D with no focal scanning. Chapter 31.5, Aberration correction in light fields re-routes recorded rays to where an ideal lens would have put them. And a closing Frontiers: I want a light-field camera weighs the standing wish against the engineering reality: the dimensionality gap and the bandwidth problem that is the field's real bottleneck.

The array end of the spectrum, which puts many apertures to work, for high-speed, HDR, super-resolution, bullet time, and the everyday multi-camera phone, has grown into its own following part, Part 32, Multi-aperture imaging.

One sentence carries the whole part: a conventional camera integrates the aperture's rays into a picture; record the rays instead and the picture becomes a choice you make later. Whether those rays come from one lens with a microlens array or from a rig of many lenses only determines how finely you can sample the four dimensions.

Contents of this part

▸ full collapsible outline of this part