34.1 Assorted pixels🔗⧉

A standard color sensor is uniform by design: every photosite is the same kind of device, and the only thing that varies across the array is which of three color filters, red, green, or blue, sits on top, tiled in the regular Bayer mosaic. That uniformity is what gives Part 15, Demosaicking and joint methods a regular sampling pattern, although missing channels still make reconstruction underdetermined (every pixel is a known sample of a known channel) and it is what lets us treat the sensor as one homogeneous measuring instrument. But it leaves a degree of freedom unused. The pixels are already individually addressable; their microlenses, filters, and even their photodiodes can be engineered pixel by pixel. So why make them all measure the same thing?

The general idea, named and formalized by Nayar and Narasimhan, is assorted pixels: deliberately give different pixels different jobs, so that a single exposure samples several properties of the incoming light at once, then reconstruct the full-resolution maps of each afterward, the way demosaicking reconstructs full-resolution color from a sparse mosaic Nayar & Narasimhan 2002 (Assorted Pixels). Bayer is then just the most familiar special case (the assorted property is wavelength, in three bins). The same machinery, interleaving a few kinds of measurement on one chip, undersampling each, and interpolating back, extends to exposure, polarization, focus, and more. Every such sensor is a single trade, made once on the silicon: spend spatial resolution to gain an extra dimension of the light.

34.1.1 Dual-pixel and phase-detect pixels: buying depth and focus🔗⧉

The most widespread assorted pixel sits in nearly every modern phone and mirrorless camera: the dual-pixel (or, more generally, phase-detection autofocus, PDAF) pixel. It supports autofocus and, as a bonus, provides a weak depth cue.

Recall that a single photosite ordinarily sits under one microlens and reports one number: the total light from the lens's whole exit pupil funnelled onto it. A dual-pixel splits the photodiode in two under that shared microlens, so the left half-diode collects light that came through the right half of the lens aperture and the right half-diode collects light from the left half (the microlens flips them). Each half therefore sees the scene from a slightly different viewpoint, forming a stereo pair with a baseline of about half the aperture, captured on one sensor with no second camera (Figure 34.1.2⛶ fullscreen).

When the scene is in focus, both halves see the same image and their two signals line up. When it is out of focus, the two half-images are shifted relative to each other, and the sign and size of that shift, the phase difference, say how far out of focus the point is and in which direction (front or back). That is the whole principle of phase-detection autofocus: the camera estimates signed defocus from the sub-image disparity and uses it to drive focus. Calibration, low contrast, noise, and subject motion can still require further corrections. Because the split is at the pixel, this happens at thousands of points across the frame at once, at video rates. This is why on-sensor PDAF, developed further in Chapter 22.9, Autofocus, displaced the older dedicated AF modules.

For the final picture, the two halves are simply summed: both halves contribute to the image. Isolation, fill factor, and read-noise differences can prevent exact equivalence to an unsplit pixel. The depth information is encoded in the difference of the two halves. A quad-pixel (2×2 on-chip lens) variant splits into four, adding vertical as well as horizontal phase so the sensor can focus on horizontal edges too.

That same left/right pair is a tiny two-view light field, and the per-pixel disparity between the two views is a low-baseline defocus cue that becomes metric depth only with optical calibration, including the lens focus setting. This is the basis of single-camera depth-from-dual-pixel, which powers phone portrait modes: learn to turn the subtle dual-pixel disparity into a clean depth map, then synthesize background blur [@garg-etal-2019; @wadhwa-etal-2018]. The same dual-pixel data also drives single-image defocus deblurring Abuolaim & Brown 2020. The catch is that the baseline is microscopic (half an aperture, millimeters at most), so the disparity is tiny and ambiguous, so learned priors do most of the work, and dual-pixel depth is a cue, not a laser range-finder.

34.1.2 Clear, white, and other color-filter variants🔗⧉

The Bayer mosaic's red, green, and blue filters work by absorption: a red filter passes red and throws away green and blue, a substantial fraction of incident photons, with the loss depending on the illuminant and filter spectra. For a large pixel in good light that loss is affordable; for a tiny pixel in dim light it is painful. So one family of assorted pixels keeps the geometry of a mosaic but changes some of the filters to gather more light.

The simplest is to make some pixels clear, using a white (W) or panchromatic pixel with no color filter, passing the whole visible band. An RGBW mosaic interleaves these white pixels among the colored ones: the W pixels collect far more photons, so they carry a higher-SNR broadband signal, which is not automatically calibrated perceptual luminance, while the sparser R, G, B pixels supply the chroma. Because the eye resolves luminance far more finely than color (the same fact that justifies Bayer's doubled green, and chroma subsampling in JPEG), spending most of the resolution on a bright luminance channel and reconstructing color at lower resolution is a sensible low-light trade. The cost is a harder, less standard demosaicking problem and a risk of the white pixels saturating first.

A related idea swaps the additive primaries for subtractive complementary filters, cyan, magenta, yellow (CMY), sometimes with green (CMYG), as used in many older video sensors. In the idealized three-band picture, each complementary filter blocks one band and passes two; actual throughput depends on the filter and illumination spectra; color is recovered by differencing. The price is noisier color (the primaries come out as differences of large numbers) and more complex processing, which is why CMY faded as sensors grew sensitive enough not to need it. The trade is the same as RGBW: clean color for photon efficiency. (The most aggressive version of this idea abandons absorptive filters entirely and routes each wavelength to a different pixel with a per-pixel nano-prism, recovering the light a filter would discard; this is covered in Chapter 34.2, Modern sensors (quad Bayer, in-sensor HDR, and beyond) and in the story of Part 15, Demosaicking and joint methods.)

34.1.3 Polarization pixels🔗⧉

Color is a property of light a normal sensor measures (in three coarse bins); polarization is one it discards entirely. Yet polarization carries information color cannot: the glare off water, glass, and paint is strongly polarized, so its polarized component can be estimated and reduced, though not every reflection is fully polarized; the polarization state of skylight encodes the sun's direction; and the way a surface polarizes reflected light constrains its orientation (shape-from-polarization) and helps tell dielectrics from metals.

The assorted-pixels way to capture it is to put a micro-polarizer over each pixel, a wire-grid analyzer etched directly onto the sensor, oriented at one of four angles, 0°, 45°, 90°, 135°, tiled in a repeating 2×2 super-pixel. The four neighboring pixels then measure the light through four polarizer orientations, from which the per-super-pixel Stokes parameters follow: the total intensity, the degree of linear polarization, and the angle of polarization. This is precisely Bayer's logic with polarization angle substituted for wavelength: four interleaved measurements, deinterleaved and interpolated to full resolution, at the cost of dividing the native resolution by four (Figure 34.1.3⛶ fullscreen). Sony's IMX250MZR brought this to a mass-produced 5-megapixel global-shutter sensor by fabricating the wire-grid polarizers beneath the microlenses, on-chip. The 3.2-megapixel back-illuminated research device in Maruyama et al. 2018 is a different sensor from the 5-megapixel IMX250MZR. Combine polarization with a color mosaic and you get a color-polarization sensor that assorts two extra dimensions at once, at the cost of even coarser native sampling of each.

34.1.4 Spatially varying exposure: assorting for dynamic range🔗⧉

The dimension a single exposure runs out of is dynamic range: the bright window clips to white while the dim interior drowns in the noise floor (Chapter 2.8, Noise, signal-to-noise ratio and dynamic range). The usual fix is to bracket several exposures in time and merge them (HDR merging), but that costs frames and invites motion ghosts. Assorted pixels offer a way to capture multiple exposures in space, in one shot.

The trick, spatially varying exposure (SVE), due to Nayar and Mitsunaga, is to give neighboring pixels different sensitivities, by laying a mosaic of neutral-density filters of varying transmittance over the array, or by giving interleaved groups different integration times. Different readout gains can protect downstream headroom, but cannot undo photodiode saturation and are not equivalent to collecting fewer photons Nayar & Mitsunaga 2000. Where a region is bright, the low-sensitivity pixels stay unsaturated and record it; where it is dark, the high-sensitivity pixels lift it above the noise. Within the covered radiance range, and when a neighborhood samples comparable scene content, at least one sensitivity can be well exposed, so a single frame holds the whole range, and a reconstruction step (again, demosaicking's cousin: deinterleave the exposures, discard the saturated and the buried, interpolate) yields one high-dynamic-range image. The cost is the familiar one: resolution is traded for range, and the merged image is softer wherever it had to fall back on the sparser well-exposed pixels.

This idea is the conceptual seed of the in-sensor HDR schemes that ship in phones today, such as staggered long/short rows and quad-Bayer cells read at two exposures, which are really SVE moved onto the pixel array and refined; we follow them in Chapter 34.2, Modern sensors (quad Bayer, in-sensor HDR, and beyond). The split-pixel or lateral-pixel designs are a hardware cousin: a single pixel is divided into a large, sensitive sub-photodiode and a small, low-sensitivity one (sometimes with its own tiny in-pixel storage), so the same location is captured at two sensitivities at once, assorting for dynamic range within a pixel rather than across neighbors, and sidestepping the spatial-resolution loss in exchange for added pixel complexity.

34.1.5 The common thread🔗⧉

These sensors all implement the same basic idea. A dual-pixel assorts for viewpoint (hence focus and depth); RGBW and CMY assort for photon efficiency (more light, reconstructed color); the polarization mosaic assorts for polarization angle; spatially varying exposure assorts for sensitivity (hence dynamic range). In each, the sensor undersamples several signals on one interleaved grid and a reconstruction step, a direct descendant of Bayer demosaicking, deinterleaves and interpolates them back to full resolution. You gain a dimension the uniform sensor threw away, at a cost in resolution (or pixel complexity). The rest of this part keeps asking the same question, which dimension is worth a sensor's pixels? It asks it of ever more exotic pixels: ones that time individual photons, count change events, or measure velocity directly.