15.12 Hardware alternatives to the mosaic🔗⧉

A color filter array is a decision to sample color in space: three kinds of measurement interleaved across the sensor, one per pixel, with the missing two thirds reconstructed by everything in the preceding chapters. It is not the only place to put that sampling.

You can put it in time, taking several exposures with different filtration. You can put it in depth, stacking photodiodes so one pixel measures three things at once. You can spend spatial resolution on a second dimension of information instead of on color. Or you can add bands the eye does not have.

The useful choice depends on motion, required color accuracy, light level, size and cost. A long astronomical exposure can still contain tracking motion, atmospheric variation and moving objects; a phone can also capture a longer static sequence.

15.12.1 Pixel shift: spending time, precisely🔗⧉

In an ideal static scene, a four-shot cycle with one-pixel offsets can sample each scene-grid position through red, both green phases and blue. This avoids interpolating missing color components at covered interior positions. Actual fusion still needs calibration, accurate offsets, valid boundaries and motion handling.

Two patterns are in use and they buy different things. A one-pixel, four-shot cycle buys color: full color at the sensor's native resolution, direct sampling of each color at covered positions, with demosaicking removed from the pipeline entirely. A half-pixel cycle, usually eight or sixteen shots combining whole- and half-pixel steps, buys resolution as well, interleaving the sampling grids finely enough to raise the effective pixel count, which is the deliberate version of the hand tremor that Chapter 14.2, Multi-frame super-resolution exploits. Sony calls it Pixel Shift Multi Shooting, Pentax calls it Pixel Shift Resolution, OM System calls it High Res Shot, and Hasselblad calls it Multi-Shot, which tells you the market: landscape, product, and cultural-heritage work.

Commanded actuator offsets do not guarantee exact scene registration. Shake, lens distortion, actuator error and subject motion can perturb them. Fusion may reject moving regions or use a single-frame reconstruction there. Controlled pixel shift and unconstrained handheld fusion therefore share sampling principles but need not use identical algorithms.

15.12.2 Foveon and stacked photodiodes: spending silicon depth🔗⧉

Silicon absorbs short wavelengths near its surface and long wavelengths deeper in, so three photodiodes stacked at different depths in the same pixel respond to three different, broad, overlapping spectral bands. That is the Foveon sensor (Lyon & Hubel 2002 (Foveon)): three measurements at every pixel location, no filter array, no demosaicking, with reduced CFA-specific color aliasing when all channels share the same grid. Ordinary spatial aliasing and channel-response differences remain; some stacked designs also use different sampling densities across layers. The full account of the physics is in Chapter 4.6, Sensing color: multiplexing strategies; the reason it belongs here is what it implies for reconstruction, which is that missing-color interpolation can be reduced while spectral separation and noise estimation remain necessary.

It did not displace Bayer, and the reasons are instructive because none of them is about the idea. The three depth responses overlap heavily and are not the primaries anyone would choose, so recovering colorimetric values needs a color-separation transform whose coefficients and conditioning can amplify noise: the color separation is set by silicon's absorption curve rather than by a filter someone designed. Carriers generated at one depth diffuse to another, which is crosstalk that further mixes the bands. The potential advantage is collecting useful light at several depths without a surface color mosaic. Not every incident photon is collected, however, and collection efficiency differs by layer. The resulting benefit depends on both collection and how well the overlapping responses can be separated.

The lesson generalizes past this one sensor. A measurement that captures more per pixel is only better if the mixing it introduces can be inverted without amplifying noise, which is the same conditioning argument that governs every multiplexed measurement in Part 34, Computational sensors.

15.12.3 Assorted pixels: spending spatial resolution on another axis🔗⧉

A mosaic spends spatial resolution to sample color. Nothing says color is the only thing worth interleaving. Give neighboring pixels different exposures and one capture carries high dynamic range, which is spatially varying exposure. Give them different spectral filters, more than three, and one capture carries a coarse spectrum. Give them different polarizer orientations and one capture carries polarization.

This is the assorted-pixels idea (Nayar & Narasimhan 2002 (Assorted Pixels); the exposure case is Nayar & Mitsunaga 2000), and it is the general form of which the Bayer mosaic is one instance: an array in which the per-pixel measurement varies according to a pattern, and reconstruction demultiplexes it. Everything the demosaicking chapters said transfers, including the frequency reading, since a periodic assortment modulates its extra dimension onto carriers exactly as the color mosaic does, and the reconstruction is again a demodulation whose limit is set by how much the bands overlap.

The full treatment is Chapter 34.1, Assorted pixels in Part 34, Computational sensors, and the high dynamic range case is Chapter 34.3, On-sensor HDR. What matters here is the framing: once you see the mosaic as one multiplexing scheme among many, the question stops being how to demosaic and becomes what to multiplex.

15.12.4 RGB-NIR: adding a band the eye does not have🔗⧉

Silicon is sensitive well past the visible, out to roughly 1000 nm, and a normal camera throws that away with an infrared-cut filter because near-infrared light contaminates color. An RGB-NIR array instead keeps it, replacing one filter of the tile with one that passes near infrared, or interleaving a fourth filter type into a larger tile.

Near-infrared measurements can distinguish materials that look similar in visible light. Vegetation reflectance and active infrared illumination are useful examples. Haze penetration depends on wavelength and particle size; it does not guarantee sharper detail or make material discrimination universally easy.

What it costs is a visible color channel and a harder reconstruction. The array now samples four bands where it sampled three, so each is sparser; the color filters themselves leak into the infrared, so the visible channels are contaminated and must be corrected using the infrared measurement; and the fusion of a visible image with an infrared one is its own problem, since a scene can be high contrast in one and flat in the other. The reconstruction is demosaicking with an extra band and a cross-band correction, which is why it belongs in this part rather than in the sensor part.

15.12.5 Color wheels: spending time, sequentially, for many bands🔗⧉

The oldest answer of all is to take three photographs through three filters, one after another. Prokudin-Gorsky's color photographs of the Russian Empire were made this way in 1909, and the technique never left the places where the scene holds still.

Astronomical filter wheels expose the full sensor through a chosen spectral filter. Every pixel samples the same band, avoiding a spatial color mosaic, but the filter still rejects wavelengths and each band receives only its share of total exposure time. Narrowband filters can isolate emission lines; standardized filter systems support calibrated photometry when their full responses and observing conditions are accounted for.

The cost is time and motion, in that order. A sequence of exposures is only a color image if nothing moved between them, and where something did move, an aircraft, a satellite, a planet's rotation, the frames disagree and the disagreement is visible as colored fringing. The remedies are the ones this book has already given: register before combining, and combine robustly.

Push the number of filters up and the wheel becomes a spectrometer. Ten bands, fifty, two hundred, and each pixel carries a spectrum rather than a color, which is Chapter 34.11, Hyperspectral imaging, color wheels. The progression is a straight line from the Bayer mosaic: three bands sampled in space, three sampled in time, many sampled in time, and finally many sampled by a mechanism designed for the purpose. What changes along it is not the mathematics of reconstruction, which stays a demultiplexing problem, but what you are rich in and what the scene will hold still for.

Astronomical filter imaging illustrates what sequential color buys. With a monochrome detector behind one chosen filter, every photosite samples that band; there is no missing-color mosaic to interpolate within the exposure. A narrowband filter can isolate emission around hydrogen alpha, oxygen III, or sulfur II, so combining bands can depict different physical emissions rather than approximate ordinary camera RGB. Standard photometric passbands also support measurements that can be calibrated and compared across instruments. The filter still rejects other wavelengths, and separate exposures spend observing time on separate bands. This is useful for sufficiently stable targets, not a way to collect every wavelength simultaneously.

Johnson-Cousins and Sloan passbands are examples of standardized astronomical systems. Their purpose is not simply to make pleasing colors: an observation can be calibrated into a defined band measurement and compared with observations elsewhere. Atmosphere, detector response and the actual filter transmission still enter that calibration. Choosing bands for physical measurement is a different goal from choosing three broad responses for a familiar-looking photograph.

15.12.6 Four ways to multiplex color🔗⧉

The mosaic is a clever answer to a hard constraint (a photosite measures one number, and color needs three) but it is not the only answer, and seeing the alternatives makes clear what the mosaic trades away. Broadly, you can multiplex the three measurements in time, in space, or in depth.

• Temporal multiplexing: take three exposures through three filters, one after another. Sequential filtered acquisition was used by Sergei Prokudin-Gorskii for color photographs of the Russian Empire around 1900, shooting three plates through red, green, and blue filters. (The same idea lives on in the color wheel spun in front of a monochrome sensor.) You get three real measurements at every pixel and need no interpolation, but only for a static scene; anything that moves between exposures shows color fringes.
• Spatial multiplexing: one measurement per location, varying the color across the grid: the Bayer mosaic itself, and, not coincidentally, the strategy your retina uses (a mosaic of long-, medium-, and short-wavelength cones). Single sensor, high resolution, mature technology, at the cost of interpolation (this whole chapter), color jaggies, the resolution lost to the OLPF, and the light absorbed by the filters.
• Three sensors: split the incoming light with a prism and route it to three separate sensors, one per color (the 3-chip / 3-CCD video cameras, named after the charge-coupled device (CCD) sensor). Three real values per pixel and almost no photons wasted (the prism redirects rather than absorbs), but you pay in cost, bulk, and alignment.
• Depth multiplexing: stack the color-sensitive layers so each pixel measures all three at once at different depths: color film (tripack emulsions) and the Foveon sensor, which exploits the fact that longer wavelengths penetrate silicon more deeply. Three numbers per pixel and good light efficiency, but it needs more color processing and tends to be noisier.

These all tie back to Chapter 4.1, Linear color reproduction and CIE XYZ of the book; the takeaway is that the Bayer mosaic wins on cost, resolution, and single-sensor simplicity, and demosaicking is the computational price we pay for that win.

15.12.7 Routing color instead of filtering it🔗⧉

Absorptive color filters discard out-of-band light. The loss depends on illumination, filter spectra, and sensor response; “two thirds” is only a simplified equal-band model, not a universal Bayer sensitivity penalty.

Color routers redirect wavelength bands toward selected sites rather than relying only on absorption [@nishiwaki-etal-2013; @zou-etal-2022]. They can improve collection efficiency, but transmission loss, spectral leakage, incidence angle, and spatial coupling remain design constraints. A Bayer-like output still needs calibration and a reconstruction model suited to its response.

Spectral separation also permits computational optics that focus bands at different depths Cossairt & Nayar 2010 (spectral focal sweep). Such systems need assumptions relating scene content across wavelengths; narrowband illumination or strongly colored materials can violate them. The optical-design chapters develop these tradeoffs. Equal photon totals alone do not guarantee equally informative color or depth measurements.

A pixel-scale color router uses wavelength-dependent optical structures to send light laterally toward appropriate sites. The intended red light that would be absorbed over a green-filtered site can instead contribute to a neighboring red measurement. The useful gain is the measured throughput of the complete system, including unwanted spectral mixing, rather than the disappearance of every loss.

Dispersion can also be used deliberately instead of corrected. If different wavelength bands focus at different depths, a single capture contains differently blurred spectral views. Summing those bands on a monochrome detector gives one combined blur; measuring them separately provides several observations with different focus behavior. Recovering depth or an extended-depth image then depends on relating content across the bands. A sharp red feature does not reveal an unmeasured blue feature merely because both belong to the same object. This connects the color-sampling problem to computational optical design and its photon budget.

15.12.8 One table, one question🔗⧉

Mosaics are compact single-sensor solutions. Stacked sensors, assorted-pixel arrays and multi-sensor spectral splitters can also acquire color in one exposure. The comparison concerns cost, throughput, sampling and conditioning, not exclusive access to single-shot capture.