31.10 Practical aspects of light field cameras🔗⧉

I want a high-end, interchangeable-lens light-field camera, and I want it now. I have wanted one for twenty years. Everything the last several chapters described, such as refocusing after the shutter, dialing the aperture in software, nudging the viewpoint, and correcting the lens's own aberrations by re-routing rays, is real, demonstrated, and sitting in research labs and a few discontinued products. It is L13.1show carried all the way down to the optics: stop throwing away the cone of rays a pixel integrates, keep each ray, and defer several optical decisions out of the moment of exposure. That is the dream, and it is a good one. The annoying part is that it has not won. There is no light-field option on the camera I actually own, and the one consumer product that tried, the Lytro, is gone. This closing chapter is about why, and it is, fair warning, more opinionated than the rest of the book, because the obstacle is not a mystery of physics. The basic optical operations have been demonstrated. My main concern is the plumbing for their data, alongside the resolution, calibration, and lens-integration work a product still needs.

Let me take the two obvious objections, resolution and bandwidth, in order, because they dominate the camera I have in mind, and the second one is the real one.

31.10.1 Do you lose all that resolution?🔗⧉

But the resolution, you tell me. A plenoptic camera puts a microlens array just in front of the sensor (see Chapter 31.2, Light field cameras), and under each microlens the sensor spends a little patch of pixels recording which direction the light came from rather than just how much of it there was. Spend, say, a 10×10 block of photosites on angle, and a 40-megapixel sensor seems to deliver a 0.4-megapixel picture. Stated that way it sounds fatal, and it is the first thing everyone says.

The short answer is that the naive ratio overstates the loss. The reason is a dimensionality gap. The light field of a real scene is a 4-D object, but it is nowhere near full-rank because most of its angular variation is low-frequency and highly redundant. Look at the angular dimension at a fixed point: as you walk your eye across the aperture, what one scene point sends you barely changes. Most surfaces are close to Lambertian, so they look about the same from every aperture direction; the angular signal only carries real information at depth edges, specular highlights, and occlusions. In other words, the full 4-D field a plenoptic camera samples is far from arbitrary and lives near a much lower-dimensional set than its pixel count suggests. So the effective resolution loss is smaller than the brute ratio of angular samples implies, and clever reconstruction (super-resolution that borrows from the redundant angular samples) claws a good deal of it back. You do pay. You do not pay the full naive factor. That phrase, the angular dimension is mostly redundant, is also the key to the real problem.

It helps to put numbers on it, and the numbers have moved in the light field's favor. Start from a 200-megapixel sensor; phones already ship 200 MP. Spend a 5×5 microlens subdivision on angle and you still keep a clean 8 megapixels of 2-D image with no reconstruction tricks at all, and a hoped-for roughly 32-megapixel reconstruction if depth and sub-view information support that gain. The latter is an engineering target, not a result guaranteed by the pixel count. For that you gain about 5× in depth-of-field and refocus latitude: you can shoot at f/1.4 (all the light of a fast lens) yet refocus within the depth of field of f/7. Now project the same trend forward. If a phone can reach 100 MP, a full-frame sensor can reach 500 MP; a 9×9 subdivision gives about 6.2 million basic spatial samples. My targets of roughly 20 megapixels with reconstruction, or 50 with stronger learned assumptions, require gains beyond that raw count and should not be read as measured resolving power, while buying 9× in focus latitude. And the aberration argument compounds it: because you can correct aberrations in post, and each sub-aperture only has to be sharp within its own narrow ~f/9 cone, you can build very fast f/1 lenses and shoot wide open at f/1, but enjoy the depth of field and the recoverable focus of f/9. The resolution objection, in short, gets weaker every sensor generation. (These figures are from my blog, thecomputationalphotographer.net; they are order-of-magnitude, but the trend is the point.)

31.10.2 Can you do video? Is it practical? The bandwidth problem🔗⧉

The main constraint is bandwidth. This is the objection that matters, and it is where I think the field actually stalled. The light field is not a fixed cost you pay once at the optics; it is a data rate you pay for continuously, every frame, forever. The following numbers are order-of-magnitude estimates, not spec-sheet values, and I will keep them frankly approximate.

Take a full-frame sensor, the 24×36 mm format a serious camera uses, and give it a pixel pitch of about 4.5 µm, a plausible full-frame example rather than a modern high-resolution phone pixel pitch. That is roughly 40 megapixels of raw photosites (see Chapter 2.7, Sensors — photosites, CCD vs CMOS for where pitch and read-rate come from). Forty megapixels for one frame is fine; cameras do that today. Now ask for video. At video rates you are reading those 40 megapixels tens of times a second, so the sensor is producing on the order of a gigapixel per second at plain 30 fps, and if you want the high frame rates that make a light-field camera interesting, tens of gigapixels per second. At 12 bits a sample that is hundreds of gigabits per second coming off the sensor. And remember, a light field already makes this worse because you read a full sensor to recover a smaller image, so the data rate is, if anything, worse than these figures for the picture you actually keep.

Now look at what you can write. Fast removable storage of the kind a camera uses, for example an XQD card, sustains on the order of 400 megabytes per second, a few gigabits per second. For 40 million samples at 30 fps and 12 bits/sample, the uncompressed rate is 14.4 Gbit/s, or 1.8 GB/s, before overhead. Against this illustrative 400 MB/s write budget, the gap is about 4.5×; at 1000 fps it is 60 GB/s, a 150× gap. A hypothetical 10:1 compression would fit the 30-fps example but still leave a 15× gap at 1000 fps. These are design calculations, not a universal limit of current storage. Ordinary JPEG ratios do not establish what compression preserves the angular detail needed for refocusing. It is not the lenses, microlenses, or refocusing math. Those pieces are already solved. The bottleneck is moving and storing the data the optics happily generates.

So the open problem, the one I would actually like someone to solve, is quick, local light-field compression, cheap enough to run on the sensor or right beside it, in real time, before the data ever has to cross the storage bus. This is where the earlier section becomes useful: the dimensionality gap is exactly the thing to exploit. The angular dimension is mostly redundant, so it should compress brutally well. I have long suspected the right primitive is something like a steered DCT across the angular dimension, a transform that knows the angular samples of a given point are nearly identical and a slope (the local disparity) predicts the residual, so almost all the angular energy collapses into a few coefficients. Get that running in silicon next to the photosites and the data rate becomes a manageable stream. That, not the optics, is what stands between me and the camera I want.

One optical subtlety belongs here, because it sets the units of the data rate. The microlens sub-image under each lenslet is an image of the main lens's aperture, the diaphragm. So the diaphragm controls how big each sub-image is and therefore how many angular samples you actually get: open the main lens wide and each microlens sees a large disc, filling its patch of photosites with angular detail; stop it down and the disc shrinks, the sub-images shrink, and angular samples go unused. Overlap instead occurs when the aperture image becomes too large. The aperture you choose is not just a depth-of-field knob anymore; it is the sampling rate of the angular dimension, and it has to be matched to the microlens geometry. This is the constraint Ng and colleagues made precise Ng et al. 2005: for the sub-images to tile the sensor with neither gaps nor overlap, the f-number of the main lens must match the f-number of the microlenses. Open the main lens wider than that match (smaller f-number) and adjacent sub-images overlap and crosstalk; stop it down (larger f-number) and each sub-image shrinks, leaving rings of wasted photosites between the lenslets and throwing angular samples away. So on a fixed plenoptic camera you effectively cannot stop the diaphragm down without harming angular sampling. The aperture, the photographer's most basic control, is no longer free.

One possible fix is to let the microlens array move closer to or farther from the main sensor, to adjust the aperture-image scale. Moving the array also changes microlens conjugacy and focus, so this is an optical-design proposal, not a complete one-parameter solution. It is an obvious approach that, as far as I know, has not been built. It is one more thing that has to be right for the data to be worth its bandwidth.

31.10.3 The time dimension is high speed🔗⧉

There is one more thing you can do with all those redundant angular samples, and I find it the most satisfying trade in the whole subject: spend the angular dimension on time instead. A camera array (the grid of full cameras at the array end of the spectrum from Chapter 31.2, Light field cameras) does not have to fire all its cameras at once. Stagger the triggers. If a ring or grid of cameras fires in a tight rolling sequence, microseconds apart, you have traded the parallax between cameras for a sequence in time, and you get an ultra-high-speed capture of a single instant from a single (nearly) viewpoint. The light field becomes a light field in time. The same hardware that records all the rays can instead record all the moments, because the angular samples were redundant enough to repurpose. It is the cleanest illustration that the 4-D field is a budget you can re-allocate: angle, or time, or some of each. (For the recombination side of fast capture, turning sparse moments back into smooth motion, see Part 30, Video.)

31.10.4 Adjacent frontiers, briefly🔗⧉

A handful of neighbors are worth a pointer rather than a chapter.

Dual-pixel autofocus is the smallest possible light field. Split each photosite into two half-pixels behind a shared microlens and you have recorded exactly two angular samples, namely two half-aperture views of the scene. That is a light field with an angular resolution of two. It is plenty to do phase-detection focus (the two views shift relative to each other by an amount that tells the lens which way and how far to move; see Chapter 22.9, Autofocus) and even a coarse depth map from the tiny disparity between the halves Wadhwa et al. 2018. That depth map is what drives the portrait-mode background blur of Chapter 23.2, Fake (synthetic) depth of field. So a stripped-down, two-sample light field is used in many phone cameras, sometimes alongside other depth cues. It is not the camera discussed here, but it does show that the idea is practical at small scale and cheap at the bottom end.

Displays are the replay side. Integral photography was always a round trip: capture through a fly's-eye array, then replay through the same array so each eye sees the right elemental view. The replay half lives on in lenticular prints and integral-imaging / autostereoscopic 3-D displays, and in the on-again-off-again dream of 3-D TV, and in Google's Project Starline, which is a light-field capture-and-display telepresence booth in all but name. The light field has two ends; this part spent its time on capture, but the display end is the same object run backward.

Gigapixel arrays. Dave Brady's AWARE cameras tile a shared objective and an array of small sensors into one multi-gigapixel instrument, a different way to spend a sea of pixels, and the spatial cousin of the array light-field camera: it tiles the field rather than the aperture, but runs straight into the same data-rate wall described above. They are a scaling-laws story about the difficulty of reaching gigapixel resolution with one conventional lens, and how an architecture escapes, so the full treatment now lives in the optics part; see Chapter 22.6, Scaling laws in optics.

Meta-optics. The newest pointer: metasurface / meta-optic light-field sensors, including 2022 Nature work, that fold the angular sampling into a single flat nanostructured optic instead of a microlens array. Whether it changes the bandwidth arithmetic remains open, but it is the newest attempt on the capture side.

31.10.5 So, do I get my camera?🔗⧉

Here is the plain verdict. The core optical operations have been demonstrated. Refocusing, synthetic aperture, viewpoint shift, even computational correction of the lens's own aberrations, all solved, all demonstrated, some of it shipping in two-sample form in your pocket. What is not finished is the data plumbing: the demanding video and high-speed configurations need a data path designed around their sample rates, rather than around ordinary rendered-image storage. Close that gap with quick, local, on-sensor compression that exploits the dimensionality gap in the angular dimension, and the high-end, interchangeable-lens light-field camera I have wanted for twenty years stops being a research curiosity and becomes a product. The physics gives me reason to want it. A practical product still has to meet the resolution, bandwidth, power, and calibration budgets together.