2.4 Image measurements as integrals over the plenoptic function🔗⧉

We have seen that image formation is all about deciding which rays contribute to a point in the image. In general, a pixel value is an integral of the incoming light over aperture, sensor area, wavelength, and time. The light reaching the camera is not one ray but a whole bundle, spread over a range of directions (the aperture), a patch of the sensor (the pixel's area), a band of colors (wavelength), and a stretch of time (the exposure). The sensor adds all of it up. The rest of the chapter elaborates that statement: first by naming the full field of light a camera draws from (the plenoptic function), then by writing the measurement itself as that integral and connecting its four axes to the camera's controls.

2.4.1 The plenoptic function🔗⧉

Before we can say what a pixel measures, it helps to name everything there is to measure. Stand in a room and ask: what is all the light here, before any camera, eye, or sensor selects from it? At every point in space, light is passing through in every direction, at every wavelength, at every instant. The function that captures all of it at once is the plenoptic function (from plenus, full, and optic, literally "full optic") introduced by Adelson and Bergen.

Parameterize a single ray of light by its 3D position $(x, y, z)$, its 2D direction $(\theta, \phi)$, its wavelength $\lambda$, and the time $t$, and the plenoptic function returns the radiance carried by that ray:

That is seven numbers in and one radiance out: a seven-dimensional description of every ray of light filling all of space, in every direction and color, at every moment. It is everything there is to see, from everywhere, at once (Figure 2.4.1⛶ fullscreen).

There is a deeper lesson in Adelson and Bergen's paper that is usually forgotten: it tends to be cited only for naming the function. Their real argument was that early vision is, throughout, the measurement of the local structure of $P$, and that the same operations, such as local averages, derivatives, and oriented filters, recur along every axis: a spatial edge, a temporal event, a binocular disparity, and a motion are all the visual system taking derivatives of the plenoptic function along different dimensions. Low-level vision is essentially one kind of task, repeated across space, time, viewpoint, and wavelength. That unifying view echoes through this book on the camera side too, because many photographic scenarios are the same operation along a different axis of $P$. A light-field (plenoptic) camera samples variation across the aperture exactly as a high-speed video camera samples variation across time: both densely slice the plenoptic function along one of its axes and resolve afterwards what a conventional camera integrated away (→ Part 31, Light fields and plenoptic cameras, Multiple exposure imaging, and the capture-the-set-decide-later lesson L13.1show). Keeping the whole function in view is what makes those parallels visible.

Seven is the classical count, and it captures everything an ordinary camera responds to, but a ray of light carries one more property the standard plenoptic function omits: its polarization, the orientation in which the light's electromagnetic field oscillates (→ Polarization). For polarization, replace scalar radiance with a Stokes vector $(S_0,S_1,S_2,S_3)$ at each plenoptic coordinate. An analyzer angle can index measurements of that state, but a single extra scalar axis is not a complete parameterization of arbitrary polarization. Humans are largely blind to it, so it is easy to forget; yet it is physically there on every ray, and it is information: the sky is partially polarized in a pattern that follows the sun, reflections off glass and water are strongly polarized (which is how a polarizing filter cuts glare), and the surface orientation of an object leaves a polarization signature. Many animals read it directly (bees navigate by the sky's polarization pattern; mantis shrimp even sense circular polarization, see Chapter 3.6, Animal eyes), and polarization cameras sample it on purpose for glare removal, reflection separation, and shape-from-polarization. It is one more dimension of the light that a measurement can choose to integrate over (throwing it away, as most sensors do) or resolve (as a polarization camera does), the same capture-or-discard choice the rest of this chapter draws for direction, wavelength, and time.

Any imaging operation is the measurement of some portion or projection of the plenoptic function. Different types of cameras yield data of different dimensionality with different resolutions for the various dimensions. A color still image has two sampled spatial dimensions and usually three spectral integrals, not three individual wavelengths. Video adds the temporal dimension. A light field camera (Part 31, Light fields and plenoptic cameras) captures 4 dimensions in the space of light rays. The generality of the plenoptic function will allow us to highlight the similarity of different effects such as motion blur and depth of field.

One limitation up front. The plenoptic function describes an existing, outgoing radiance field: it takes the illumination as already baked into the scene and asks only what a sensor can capture from the light that happens to be leaving it. Yet in a large part of imaging the lighting is not a given but a degree of freedom we actively control. A flash, a ring light, a projector's structured pattern, or the many lamps of a light stage all change what the scene sends back, and comparing the results reveals shape, reflectance, and depth that no single passive capture contains. Seen that way, the plenoptic function is the scene's response to one fixed lighting condition; the richer object is the scene's reflectance field, its response to arbitrary incident light, which active methods sample on purpose. We take the illumination as fixed for the rest of this chapter, but return to it as a variable when we reach flash and lighting in the photography part (Chapter 5.4, Working with light) and, at length, in Part 35, Computational illumination.

2.4.2 The pixel integral🔗⧉

A pixel does not sample a single ray; it integrates over four groups of variables, each weighted by how the pixel responds there (Figure 2.4.2⛶ fullscreen):

1. the aperture: all the ray directions $(\theta, \phi)$ the lens admits toward this pixel, a solid angle set by the f-number;
2. the pixel's area $(x, y)$ on the sensor, a finite patch, not a mathematical point;
3. wavelength $\lambda$, weighted by the sensor's and color filter's spectral response; and
4. the exposure time $t$, the interval the shutter is open.

The spectral axis, $\lambda$, is the one this book develops furthest elsewhere: collapsing a whole spectrum onto a few numbers weighted by $r(\lambda)$ is exactly where color is born, and how a sensor is built to make that measurement in three or more channels, together with what the resulting numbers mean perceptually, is the subject of Part 3, Visual perception and color and Part 4, Color spaces and color technology, and in particular the chapter on Chapter 4.6, Sensing color: multiplexing strategies. Here we treat $\lambda$ simply as one more axis the pixel integrates over.

Schematically, the value recorded at a pixel is

the scene radiance integrated over aperture × pixel area × wavelength × time, weighted by the pixel's spectral response $r(\lambda)$ and the geometric weight $k$ that the aperture and footprint impose. Measurement is integrating the plenoptic function against the pixel's sampling kernel, and the support of that kernel along each axis is exactly a photographic knob. Because the underlying capture counts photons, this integral is linear in scene radiance, a fact we will need both for high-dynamic-range merging and for the noise analysis in Chapter 2.8, Noise, signal-to-noise ratio and dynamic range.

The four domains are the four knobs. This matters because each integration domain is governed by a control the photographer actually sets: the integral is the camera's exposure settings, written as mathematics:

• the aperture integral is set by the f-number $N = f/D$: stopping down shrinks the cone of directions each pixel sums over (less light, more depth of field); opening up widens it;
• the exposure-time integral is set by the shutter speed: a longer interval integrates over more time (more light, but more motion blur);
• the pixel-area integral is fixed by the sensor's pixel pitch (the physical size of a photosite) so larger photosites collect more photons at equal sensor irradiance and exposure, while larger sensor area does not by itself imply larger pixels;
• the wavelength integral is shaped by the spectral response (the sensor's sensitivity curve times the color filter over each photosite) which collapses a whole spectrum into a single number per channel.

The one familiar exposure control missing from this list is ISO, and the omission is instructive: ISO is not an integration limit. It scales the result of the integral after the light is collected (analog or digital gain), rather than changing how much light is gathered, which is exactly why pushing ISO amplifies noise along with signal, as the noise section will show.

The deeper reason to write capture this way is that each of the four integration domains earns its own topic later, and much of photography is the craft of choosing how to shape these four integrals. Every later sensing topic is just what happens when you change the support of one axis:

• integrating over the aperture is what produces depth of field (the lens chapter): a wide aperture integrates over a fat cone of directions, so an out-of-focus point spreads its energy across many pixels: the circle of confusion. The light-versus-blur trade lives entirely on this one axis;
• integrating over the pixel's area is sampling and antialiasing (Basic Image Processing): a finite footprint makes each sample a box-filtered average rather than a point sample, pre-blurring the image before it is laid on the grid: a wider footprint suppresses aliasing at the cost of resolution;
• integrating over time is motion blur (Motion/Video): a long exposure sums a moving scene over the interval, the temporal sibling of the aperture, trading light against sharpness;
• integrating over wavelength is color sensing: the spectral response collapses a whole spectrum into one number per channel, exactly as a cone does (Chapter 3.2, Perceptual color and trichromatic vision). Because a color-filter array gives each photosite a different $r(\lambda)$, color is spatially multiplexed across the Bayer mosaic and must be reconstructed by demosaicking.

The rest of the sensing story reduces to one question, which kernel, how wide, and what does it lose: antialiasing widens the spatial kernel, depth of field is the aperture kernel, motion blur is the time kernel, and demosaicking inverts the wavelength multiplexing. Each is also a source of noise: a narrower kernel along any axis (a smaller footprint, a shorter exposure, a stopped-down aperture) integrates fewer photons, and fewer photons mean a noisier estimate, which is exactly where Chapter 2.8, Noise, signal-to-noise ratio and dynamic range picks up.

All four axes are then summed into one number: the conventional sensor couples them, discarding how the light was distributed across directions, area, color and time alike. The ideal sensor would do the opposite: decouple the axes, measuring each one separately so we could decide after the shot how to combine them. Chasing that decoupling is a whole branch of computational photography, which the end of the chapter frames.

Framed this way, the measurement integral is a general language for imaging. Every imaging system, from a pinhole to a light-field camera, is described by which weighting it applies to the plenoptic function before integrating: the choice of weights over direction, area, wavelength, and time is the design of the system. And because integration is linear, every such system is a linear operator on the plenoptic function, the viewpoint that Chapter 6.7, Image formation as a linear system develops in full.

For the sake of simplicity, we routinely speak of imaging in terms of infinitesimal quantities: a pinhole that admits a single ray, an exposure that freezes an instant, a pixel that samples a point, a single wavelength of light. These idealizations keep the geometry clean and the diagrams legible. But no real measurement is infinitesimal. A sensor reports a number only by integrating over a finite aperture, a finite area, a band of wavelengths, and a stretch of time, because every axis of the plenoptic function must be summed over to gather enough photons to register at all. The idealized ray and the frozen instant are the limit the mathematics reaches for; the finite integral is what the camera actually delivers, and the gap between the two is where depth of field, motion blur, color mixing, and noise all live.

Off-axis ($\cos^4$) falloff. One prediction falls out of this irradiance integral. Rays reaching the corners of the frame arrive obliquely, so the same scene radiance delivers less irradiance there: the aperture-and-area geometry of the pixel integral works out to a $\cos^4\theta$ dimming toward the edges, called natural vignetting (Figure 2.4.3⛶ fullscreen). It is distinct from optical vignetting (lens elements physically blocking edge rays, covered in Optics) in that natural vignetting is a clean consequence of the pixel-integral geometry alone.

To interpret the integral as expected photoelectrons, let $r(\lambda)$ include quantum efficiency and the conversion from radiant energy to photons, $\lambda/(hc)$. If $r$ denotes quantum efficiency alone, that factor must appear explicitly. Optical transmission and the projected-area factor belong in $k$. Fixing these weights makes the expected measurement linear; the observed count is random, and saturation or later encoding breaks this simple linear relation.

2.4.3 Exposure values (EV) and stops🔗⧉

The pixel integral hands back a linear amount of light: double the aperture area or double the time and you double the number. Photographers almost never speak that way. They count light in stops, where one stop is a factor of two, because both the eye and the medium respond to ratios rather than differences (the same logarithmic story behind gamma encoding and Weber's law, Chapter 3.2, Perceptual color and trichromatic vision). A stop is the natural quantum of exposure: one doubling, one halving, one bit of light. Written on the log$_2$ scale the integral lives on, the whole messy business of exposure becomes simple addition and subtraction.

The power of the unit is that it is the same unit on every axis of the integral. One stop of aperture (a $\sqrt2$ step in the f-number, since light goes as $1/N^2$, Chapter 2.3, Lens image formation), one stop of shutter (a factor of two in time $t$), and one stop of ISO (a factor of two in gain) all change the brightness by the identical factor of two, so they can compensate one another in output brightness over an unclipped, approximately linear range. They do not preserve motion blur, depth of field, photon count or read-noise behavior. That commensurability is exactly what lets the three trade against one another, the exposure triangle: give back a stop of light on one leg, take it up on another, and the intended output brightness can remain unchanged. Many different aperture-shutter-ISO combinations therefore land on the same brightness, a redundancy photographers call equivalent exposures and rely on constantly (the faucet-and-bucket reciprocity of Chapter 5.2, Exposure and capture timing).

To name a whole family of equivalent settings with a single number, photography uses the exposure value (EV):

the base-2 logarithm of squared f-number divided by exposure duration expressed in seconds. EV describes aperture and shutter settings; relating it to scene luminance also requires ISO and meter calibration. One EV is one stop. Every $(N, t)$ pair on the same EV describes the same exposure, so a single EV names an entire diagonal of the aperture-shutter grid:

Read across: each stop you close the aperture is paid back by one stop of slower shutter, and the exposure never moves. Because EV folds the settings into one number, it doubles as a measure of scene light. At ISO 100 the same EV names how bright the scene is (sometimes called the light value): a meter's reading, $L = K N^2/(t S)$, is EV in disguise, and the old sunny 16 rule ("bright sun is $f/16$ at 1/ISO") is nothing but a printed EV chart, one EV per lighting condition.

Once you have the unit, it turns up everywhere, which is the real reason to learn it. Meters report in EV, and every metered scene is placed on the EV ladder relative to mid-gray. Exposure compensation is dialed in EV ($+1$ EV to keep a snowfield white, $-1$ for a black cat), a direct nudge along the ladder. Auto-bracketing shoots frames at, say, $-2, 0, +2$ EV to merge into one high-dynamic-range image (HDR merging). A scene's dynamic range, a sensor's, a display's, are all quoted in stops (the ratio of brightest to darkest, in powers of two). Neutral-density filters are sold in stops (an ND8 is three stops), a lens's speed is "one stop faster," flash and studio lights meter in stops, and the noise story is a stop story too: because shot noise is set by photon count, every stop less light collected costs half the photons and a predictable step of signal-to-noise ratio (Chapter 2.8, Noise, signal-to-noise ratio and dynamic range). Stops and EV are the shared logarithmic language in which exposure, metering, dynamic range, filtration, and noise are all spoken; the full exposure and metering practice built on them is Chapter 5.2, Exposure and capture timing.

2.4.4 Splitting the integral🔗⧉

The integral we have just written is the bedrock of ordinary image formation: it projects a moving, three-dimensional world, seen through a finite lens, onto a flat two-dimensional image, collapsing aperture, area, wavelength and time into a single number per pixel. But a whole branch of computational photography refuses that collapse. Rather than integrate each axis away at capture, it slices and dices the integral (measuring an axis in pieces and reuniting them afterward) to recover the information the single integral throws out, and with it the freedom to form the image a posteriori.

Each axis can be split:

• Burst imaging splits the time integral into many short exposures, so they can be aligned before they are summed, recovering sharpness a single long exposure would have smeared, and putting motion blur under control after the shot (Multiple exposure imaging).
• Plenoptic / light-field cameras split the aperture integral, recording rays by the direction they came through the lens, so depth of field and focus can be chosen after the shot (Part 31, Light fields and plenoptic cameras).
• Higher-resolution sensors split the spatial (pixel-area) integral, in the most mundane way on this list: more, smaller photosites, each averaging over a smaller footprint, so the box-filter narrows and finer detail survives the sampling (the sampling-and-antialiasing trade above). What looks exotic on the other axes is, on this one, just "more megapixels." Quad Bayer sensors make the split a choice taken after metering: the color-filter array groups photosites into $2\times2$ tiles that share one color, so in good light the four sub-pixels are remosaicked to full resolution (the integral split into four finer samples), while in dim light their signals can be binned into a coarser effective pixel, combining photons collected across the group and changing the readout and reconstruction tradeoff; some sensor designs also assign different exposures or gains within such groups for single-frame HDR. The phone sensor that advertises "48 MP" yet hands you clean 12 MP files is doing exactly this: deciding, shot by shot, whether to split the spatial integral or simply perform it (Part 15, Demosaicking and joint methods).
• Hyperspectral imaging splits the wavelength integral into many narrow bands, replacing three broad responses with more spectral measurements; the finite bands still limit spectral resolution.
• Polarization imaging splits along the polarization state, the Stokes vector that replaces scalar radiance above: filters rotated to multiple angles separate light by polarization, giving a handle on specular highlights and reflections: cutting glare, or pulling a reflection apart from what sits behind the glass.

In every case the move is the same, don't integrate yet; record the pieces and decide later: it is why so much of this book is about what becomes possible once the integral is no longer a foregone conclusion.

A camera selects a slice of all the light in the world (the plenoptic function) and integrates it over aperture, area, wavelength, and time. The device that performs that integral physically (a grid of photosites counting photons, read out as numbers, and the shutter that times them) is the subject of Chapter 2.7, Sensors: photosites, CCD vs CMOS; the unavoidable jitter in the photon count, the noise it imposes, follows after that.

Recording the pieces has costs. Each separately read exposure can add read noise, and dividing aperture or spectrum among channels reduces the photons available to each measurement. More dimensions also consume sensor area, bandwidth and storage. Recombination can recover the sum under a suitable calibrated model, but it does not automatically recover the signal-to-noise ratio or temporal coverage of one uninterrupted exposure.

2.4.5 Attempts to visualize the plenoptic function🔗⧉

Seven plenoptic coordinates, with polarization added as a vector state, exceed what one static picture can show. A page gives two spatial axes, animation a third, interaction a fourth; the rest we slice or project. So every picture of $P$ fixes all but two of its axes and plots the slice, and the only real choice is which two to keep.

The useful slices are already familiar. Hold a row of the sensor as $x$ against the aperture coordinate $u$, and a scene point becomes a line whose slope is its depth, the ray-space diagram of Chapter 2.5, Depth of field; swap $u$ for time $t$ and the slope becomes a velocity, the space-time diagram of Chapter 2.6, Motion blur. Integrating down the vertical axis collapses either slice back to the recorded row, and the blur is the shadow a slanted line casts. Other devices sample more of $P$ at once: a light field as a grid of sub-aperture views (Part 31, Light fields and plenoptic cameras), a camera on a rail stacking scan-lines into an epipolar-plane image, Adelson and Bergen's pencil of all rays through a single point. None of them shows $P$ itself; each keeps a couple of axes and sums the rest away, the same measurement the rest of this chapter is about. The clearest way to feel it is to build the smallest case by hand.

Start with a still world. Shrink it to a 2-D (flatland) scene and a flatland camera, hold time fixed, and the portion of $P$ that camera can draw from is just two-dimensional: pixel position $x$ and aperture $u$. That is the light field. Each scene point becomes a straight band in the $(x, u)$ plane whose slope is its depth, and the ordinary photograph is that plane integrated over the aperture (Figure 2.4.4⛶ fullscreen).

Add motion, and the light field gains a third axis. Each moving object fills a slanted slab in the $(x, u, t)$ volume (depth tilts it in $u$, velocity tilts it in $t$), and every visualization above is one flat cut through it (Figure 2.4.5⛶ fullscreen). An instantaneous light field is a fixed-$t$ slice, and a space-time diagram fixes $u$. An ordinary photograph integrates over both aperture and exposure time; a pixel also integrates over its finite spatial footprint.

Recap: big lessons of this chapter🔗

💡 Big lesson (L2.26) — a pixel value is an integral🔗

No pixel measures "the" value at a point in space, time, or color: it reports an integral over everything the photosite cannot tell apart. Four axes appear explicitly above: the spatial footprint (the pixel's area, plus the optical blur smeared across it), the exposure time (a window, not an instant), the spectrum (wavelength, weighted by the channel's spectral sensitivity), and the solid angle of rays the aperture admits; a fifth, polarization, is usually lumped in too. Naming all the axes at once is what makes motion blur, defocus, color, vignetting, and aliasing the same phenomenon (a finite integral standing in for an idealized point sample), and the width of the integral along each axis is a knob the rest of the book spends its time controlling.