31.8 Light field microscopy🔗⧉

A biologist watching a living cell faces a frustrating dilemma. To see the whole three-dimensional shape of the specimen, the standard tools, such as a confocal microscope or a light-sheet rig, must scan: step the focus through the sample plane by plane, or sweep a sheet of light across it, building the volume one slice at a time. That works well for a fixed, stained slide. But live specimens move and change while you scan, so the slices no longer belong to the same instant, and every extra slice is more light dumped into a sample that photobleaches and dies under the dose. The scanning microscope forces a choice between speed and depth that a living specimen will not let you win.

The escape is the one we have already met for ordinary cameras: stop integrating the rays and record them. Levoy and colleagues took the hand-held plenoptic camera of the previous chapters and dropped it onto a microscope Levoy et al. 2006. Put a microlens array at the microscope's intermediate image plane, exactly where the eyepiece would normally form the magnified image, and a single snapshot captures the specimen's entire light field. From that one exposure you can refocus through the sample, look at it from oblique angles, and, with a little more work, reconstruct a sharp 3-D volume, all after the shutter, from one exposure interval. The specimen can still move during that interval, and achievable resolution depends on photon counts and reconstruction conditioning.

It is one idea you have already met: the plenoptic camera, now pointed down a microscope. We will build the optical setup, see what the spatial-versus-angular trade buys (a coarse 3-D volume), recover the volume two ways (cheap synthetic refocusing and sharper 3-D deconvolution), and close on why single-shot 3-D is the point, not a bonus.

31.8.1 The optical setup: a microlens array at the intermediate image plane🔗⧉

A microscope already does most of the work. The objective collects light from the specimen and, together with the tube (relay) lens, forms a magnified real image at a fixed plane inside the instrument, the intermediate image plane, the plane your eyepiece would relay to your eye. In an ordinary digital microscope a sensor sits at that plane and records a magnified image focused at one plane, generally with contributions from out-of-focus depths as well. Every ray that left a given specimen point through the objective's aperture lands on the same sensor pixel; the pixel is the integral over the aperture, and the angular information, namely which ray came from which part of the objective lens, is thrown away. That is exactly the integral the light field refuses to take.

So we modify the back end in precisely the way the plenoptic camera taught us. Place a microlens array at the intermediate image plane, and put the sensor one focal length behind it, at the lenslets' back focal plane (Figure 31.8.1⛶ fullscreen). Now each microlens sees one small patch of the magnified image, and because the sensor sits at the lenslets' focal plane, each microlens images the objective's aperture onto the cluster of pixels beneath it. The pixels under one microlens therefore record, separately, the rays that arrived from different points across the objective lens, that is, the angular breakdown of that image patch. A microlens plus its pixel cluster is one macro-pixel; reading off one pixel position from every macro-pixel gives one oblique view of the specimen, the view seen from one point of the objective's aperture. The single snapshot is the 4-D light field $L(u,v,s,t)$ of the specimen, with $(s,t)$ the lateral position selected by which microlens and $(u,v)$ the angle selected by which pixel under it.

31.8.2 The spatial-versus-angular trade: a coarse 3-D volume🔗⧉

We have, as always, bought the angular samples with spatial ones. The microlens array splits the intermediate image into one sample per microlens rather than one per sensor pixel, so the lateral resolution of any single recovered view drops by roughly the number of pixels packed under each lenslet. That count is exactly the number of angular (directional) samples you gain. For example, if each microlens covers a cluster of about $14 \times 14$ pixels, you record on the order of that many distinct directions through the objective, and you pay for it with a basic view with about 14 times fewer samples along each lateral axis, or about 196 times fewer samples in area. Sample count and optical resolving power are not identical. This is the same spatial↔angular trade-off the plenoptic camera made in Chapter 31.2, Light field cameras, read here in microscope units.

What those angular samples buy is the third dimension. Because each direction through the objective sees the specimen from a slightly different vantage, the recorded directions disagree about the lateral position of any point that is out of the focal plane, and that disagreement is depth. The light field therefore encodes a coarse 3-D volume of the specimen in a single frame: coarse laterally, because we spent resolution on angle, but genuinely three-dimensional, where an ordinary wide-field micrograph integrated depth contributions without enough angular information to separate them as readily.

31.8.3 Recovering the volume: synthetic refocusing, then 3-D deconvolution🔗⧉

There are two ways to turn the recorded light field into a volume, and they are the two we already know.

The cheap one is synthetic refocusing, exactly the shift-and-add of Chapter 31.4, Refocusing and synthetic aperture. To focus at a chosen depth, shift each oblique view by an amount proportional to its angular position, namely the shift that cancels that depth's disparity, and sum the views. Sweep the shift and you synthesize a whole focal stack through the specimen from the one capture, and, as a bonus, oblique perspective views of the sample come for free: pick a single direction out of each macro-pixel and you are looking at the specimen from that point of the objective's aperture, a small parallax you can rock back and forth. No part of the microscope moved.

Shift-and-add is fast but optically naive because it simply re-sums blurred views, so the refocused slices are soft and still contain out-of-focus haze from other depths. The sharper route treats reconstruction as what it actually is: a linear inverse problem. Each voxel of the 3-D specimen, glowing on its own, would produce a specific, computable pattern across the whole light-field sensor, the light-field point-spread function (PSF). Because emission is linear, the recorded light field is the sum of every voxel's pattern weighted by that voxel's intensity: a single big linear map $L = A\,\mathbf{x}$ from the unknown volume $\mathbf{x}$ to the measured light field $L$, with $A$ built from the calibrated wave-optical PSF. Diffraction is central at microscope scales; geometrical ray shifts alone do not describe the measurement adequately. Recovering $\mathbf{x}$ is then a deconvolution: invert that map, with a non-negativity constraint and a regularizer, exactly the machinery of Chapter 6.7, Image formation as a linear system Broxton et al. 2013. The payoff is a deconvolved 3-D volume markedly sharper than the shift-and-add stack, with the cross-depth haze largely removed and much of the lost lateral resolution recovered, because the inverse problem un-mixes the directions that shift-and-add merely averaged. (As always with an inverse problem, you correct only what the forward model $A$ captures, and a faithful PSF must fold in the objective's own defocus and aberrations from Part 22, Compound lenses, and aberration correction.)

31.8.4 Why single-shot 3-D is the whole point🔗⧉

Everything above earns its keep for one reason: the volume comes from one exposure. A scanning confocal microscope or a light-sheet rig builds a stack by moving (stepping the focus or sweeping a sheet), which takes time and pours light into the sample slice after slice. Light-field microscopy has no focal scanning and no moving parts: a single snapshot, and the 3-D information is already in hand. That changes what you can image. Live, moving specimens are captured at one instant, so the slices belong to the same moment and motion no longer smears the stack. A single exposure removes sequential scanning, but lower dose does not follow automatically. Compare excitation dose at matched volume rate, resolution, and signal-to-noise ratio; light-sheet illumination can be very dose-efficient. Photon sharing among reconstructed voxels and background can make light-field reconstruction demanding.

This enables fast volumetric imaging, most pointedly neural-activity imaging: a fluorescent indicator brightens when a neuron fires, and to read out a behaving brain you need the whole volume at video rate, at a volume rate adequate for the biological dynamics. Capturing the light field of the neural tissue in single snapshots lets the entire volume be reconstructed frame by frame, so the activity of many neurons across depth is recorded at once Prevedel et al. 2014. The trade is the familiar one: you accept the coarser lateral resolution of the spatial-versus-angular split, and you lean on the deconvolution to win some of it back, in exchange for seeing, in time, a living three-dimensional process without the inter-slice time offsets of sequential acquisition.