31.9 Light field networks🔗⧉

The light field promised that recording rays would make any photograph a projection you compute afterward, and it came with a large cost. The 4-D field is enormous: a useful capture is a dense grid of views, megarays of data, and rendering a new viewpoint means looking the rays up and interpolating between the ones you actually measured. Chapter 31.1, Light fields 101 ended on exactly that catch: the ray you want almost never sits on a stored sample, so you guess between neighbors, and when the angular sampling is coarse the guess shows up as ghosting. Chapter 31.7, Lumigraph and shape priors for sharper light field rendering improves the guess by feeding in geometry: reproject each ray onto an approximate depth surface before you blend, and a handful of views renders sharply. There, geometry is a prior: knowledge about the scene that buys back the samples you did not take.

This chapter keeps the goal and changes the machinery. Instead of storing a sampled table and interpolating it, learn a continuous function from a ray to its radiance. Give a network a ray, four numbers in the same two-plane indexing used for $L(u,v,s,t)$, and have it return the color, fitting the function to the views you did capture. The stored array disappears; what remains is a set of network weights, and rendering is a function call, not a lookup. And the interpolation that used to need either dense sampling or hand-built geometry is supplied by a fitted function and its inductive biases. A network trained across scenes can learn a scene prior; fitting one scene from scratch does not automatically provide that population knowledge. That learned filling-in is the crux, and it is the same move the book has made before with denoisers and deblurrers: a hand-designed prior swapped for one fit to data L7.6show. Here the hand-designed prior is the lumigraph's geometry; its learned replacement is the network.

31.9.1 NeRF: a radiance field rendered by volume integration🔗⧉

A central example of learned scene representation is NeRF, neural radiance fields (Mildenhall et al. 2020). The idea is to model the scene not as a surface but as a fog of colored, semi-transparent matter filling space, and to learn that fog as a function. A single small network $f_\theta$ takes a 3-D point $\mathbf x=(x,y,z)$ and a viewing direction $\mathbf d$, five numbers in all, and returns two things: a direction-dependent color $\mathbf c(\mathbf x,\mathbf d)$ and a direction-independent density $\sigma(\mathbf x)$ in the original NeRF formulation. The color is direction-dependent, which is what lets a learned point glint differently as you move (the angular variation a single texture cannot hold, from Chapter 31.7, Lumigraph and shape priors for sharper light field rendering); the density says how much the point blocks and emits light.

That is only half of NeRF. A radiance field is not yet an image. To get a pixel you must render the field, and NeRF renders by the physics of a participating medium. For each output pixel, shoot the camera ray out into the scene, sample many points along it, evaluate the network at every one, and integrate color front-to-back, each sample weighted by how much light survives the accumulated density in front of it. In words: walk the ray, ask the network "what color and how opaque are you here?" at point after point, and composite the answers the way fog occludes what is behind it. The integral is the classical volume-rendering equation, and it is differentiable, so the whole pipeline trains by gradient descent: render the pixels of the known views, compare to the photographs, and push the error back into the weights until the field reproduces them. Nothing supervises geometry directly: the shared density field is constrained by reproducing captured views, but multiple geometries can fit sparse images and spurious density can remain.

NeRF can reproduce convincing parallax and view-dependent appearance, but geometry, reflections, and thin structures are not guaranteed correct. But read the cost off the algorithm: every pixel requires many network evaluations, dozens to hundreds, one per sample along its ray, because the color of a pixel is an integral and you approximate an integral by summing samples. A radiance field is the right physical model, and it pays for that fidelity in forward passes. For the light-field part the relevant question is the one Chapter 31.4, Refocusing and synthetic aperture kept asking: can we get the new view in one operation instead of an integral over the ray?

31.9.2 Light field networks: a ray straight to color, in one evaluation🔗⧉

Light field networks (Sitzmann et al. 2021 (LFN)) answer yes, by going back to the light field's own definition. A light field already is the function "ray $\to$ color": that is what $L(u,v,s,t)$ means. NeRF integrates along the ray because it models the volume the ray passes through; but the thing you ultimately want, the radiance arriving along that ray, is a single number you could name directly. So a light field network drops the volume and learns that direct map: a network $f_\theta$ takes a parameterized ray $\mathbf r$, such as the four-number two-plane index or an equivalent ray encoding, and returns its color $\mathbf c$ in one forward pass. No samples along the ray, no integral, no density. Rendering a pixel is one evaluation.

Read the two side by side: NeRF's pixel is an integral of many evaluations along a ray; the light field network's pixel is one evaluation of the ray itself. This is the neural analog of the two-plane light field exactly, a continuous function over ray space, but where Chapter 31.1, Light fields 101 stored that function on a grid and interpolated it, the network is the function and evaluates it at queried rays without interpolating an explicit stored grid. Pixel-footprint antialiasing is still needed. And the network is doing the lumigraph's job: trained on sparse views, it has absorbed a learned shape-and-appearance prior and uses it to interpolate the rays nobody captured, doing in the weights what geometry once did by hand (Figure 31.9.1⛶ fullscreen; Figure 31.9.2⛶ fullscreen).

The tradeoff is clear. Because NeRF integrates a single underlying 3-D field, its shared density imposes a useful 3-D constraint across views. Direction-dependent color and underconstrained geometry still allow physically incorrect explanations of the training images. A light field network queries rays through shared weights without NeRF’s explicit volume, so it trades that particular 3-D structural constraint for the speed of single-evaluation rendering. Sitzmann and colleagues recover much of the consistency by giving the network a structured latent that ties the rays of a scene together, and by meta-learning a prior across many scenes so that a new scene is captured by a compact code rather than retrained from scratch. The tradeoff stands: a light field network renders in one pass but gives up some consistency, whereas NeRF evaluates a shared volume with many samples per ray. It is the same speed-versus-guarantees axis that ran through the Fourier-slice shortcut, now between two learned representations.

31.9.3 A family of neural light fields🔗⧉

Light field networks sit in a small family that all use the same idea: represent the field as a function you evaluate, not a table you store. They differ in what the function takes and how its consistency is enforced. X-Fields (Bemana et al. 2020 (X-Fields)) learn an implicit map from a low-dimensional coordinate (viewpoint, light position, time) to the corresponding image, using learned correspondence fields to warp and combine a sparsely captured set of view-, illumination-, and time-varying images; it is the same "learn the continuous field behind a sparse capture" idea, generalized off the pure-viewpoint axis to relighting and animation. Attal and colleagues (Attal et al. 2022) study neural light fields head-on, with ray parameterizations and network structure chosen to recover the multi-view consistency that a naive ray$\to$color map gives up, making explicit the speed-versus-consistency trade above. One strategy within this family is a meta-learned prior over scenes: train across many scenes so the network learns not one light field but the space of plausible light fields, and a new capture is then a short fit (a code, or a few gradient steps) inside that learned space. That phrase, "a prior over scenes, adapted per capture," is exactly where the next section, and the rest of the book, pick up.

31.9.4 Test-time training: the network as the per-scene prior🔗⧉

Several methods here share an adaptation step at inference: it adapts a network to one specific scene from a handful of its views. NeRF fits weights to one scene's photographs; a light field network fits a per-scene code inside a meta-learned prior; the lumigraph, the classical ancestor, fit an approximate geometry to one scene's views. In all three the object specialized to the scene at capture time, whether geometry, code, or weights, is the per-scene light-field prior. That observation has hardened into an explicit principle in recent novel-view-synthesis work that adapts the network at inference, per scene, a form of test-time training, where the model continues to learn from the test instance itself rather than being frozen after pretraining. The recent "test-time training done right" line applied to view synthesis is the modern continuation of this chapter's thesis: a network pretrained across scenes carries a strong prior, and a short test-time adaptation on the captured views specializes it to this scene before rendering, the learned successor to fitting the lumigraph's geometry, now fitting the network. Per-scene fitting from scratch, latent-code inference in a pretrained model, and updating pretrained weights at test time are distinct procedures; state which parameters change and which observations supervise them. This closes the arc of the whole part: from a stored table, to a table steered by hand-built geometry, to a network whose weights are the geometry, fit to each scene as it arrives.

31.9.5 Hand-off: from rays to radiance fields and generation🔗⧉

This chapter is the seam. Behind it lies the classical light field (capture, index, refocus) and the lumigraph's lesson that a prior buys back samples. Ahead lie two larger stories that grow straight out of it. The radiance-field idea, sketched here only as NeRF-the-context, becomes its own subject in Chapter 36.7, Photos → radiance fields and Gaussian splatting (NeRF, 3DGS): how learned volumetric scenes are built, accelerated (the many-evaluations cost is the thing everyone attacks), and turned into the workhorse of 3-D reconstruction. And the move from fitting a scene to generating one means sampling plausible scenes and views rather than reproducing measured ones, and it is the subject of Chapter 7.4, Generative AI and diffusion and the 3-D part, where the learned prior stops merely interpolating captured rays and starts inventing them. Both inherit this chapter's frame: a camera takes a scene to an image by a projection (a forward image-formation model, linear in ray radiance for fixed optics, though NeRF rendering is nonlinear in its unknown density), and a learned scene representation is precisely a function trained to invert that projection, to answer "what scene would have produced these views?" so that new views can be rendered from it. The light field network is the cleanest statement of that frame on the light-field side: the network is the scene, the ray is the query, and a photograph is one forward pass away.