Lecture Notes 5 — Demosaicking🔗⧉

Multiplexing in time🔗⧉

Shoot red, green, and blue frames sequentially through colored filters, or a rotating filter wheel. This is the principle behind Maxwell’s early color-photography demonstration (Maxwell 1861) and how Prokudin-Gorskii (Prokudin-Gorskii) captured the Russian Empire in three exposures. It is still how flatbed scanners, astronomy and fluorescence-microscopy filter wheels, telescope narrowband imaging, sequential-color machine-vision cameras, and many multispectral instruments work, because the same mechanism extends naturally from three filters to tens of narrow bands.

The implementations differ mainly in what does the sequencing. The oldest is a rotating filter wheel (or three separate plates) in front of a single sensor: three exposures through red, green, and blue filters, combined afterward. Maxwell’s 1861 tartan-ribbon demonstration and Prokudin-Gorskii’s later survey are early examples. Astrophotography still leans on it heavily: a cooled monochrome sensor (a mono sensor avoids the color filter array's light loss and keeps full resolution) sits behind a motorized filter wheel and stacks many long sub-exposures through each filter, red / green / blue for natural color or narrowband hydrogen, oxygen, and sulfur filters for nebulae, with tracking and registration to account for camera and celestial motion. Many flatbed scanners also scan spatially over time: a sensor bar is swept mechanically down the page, which is itself how a one-dimensional line sensor becomes a two-dimensional image (scanning spends time to buy the spatial axis the sensor lacks), and color can come from sequential illumination during a scan, separate passes, or spatially offset filtered sensor rows. Field-sequential color television and some endoscopes ran the same spinning-color-wheel trick in real time.

The trade-off is discrimination in time: the method assumes the scene and camera hold still across the exposures. For a static scene, each band uses the full sensor grid. The filters still reject light, and a fixed total capture time must be divided among bands. If the scene moves, its changing parts splits into colored fringes (Figure 4.6.6⛶ fullscreen), and every temporal system must first register the plates to undo the small shifts between them (Figure 4.6.5⛶ fullscreen), the simplest image-alignment problem in the book.

Multiplexing across separate sensors: the beam-splitter🔗⧉

Send the whole image to more than one sensor and give each its own color. A dichroic prism block reflects each band onto its own chip (Figure 4.6.10⛶ fullscreen): the 3-CCD / 3-chip design long standard in broadcast video cameras, three-chip cinema and telecine cameras, and scientific and machine-vision multi-sensor rigs. Because the dichroic coatings split the light rather than absorb it, the separation can retain more useful light than absorbing filters, and each channel has its own full sensor grid. (It is a dichroic beam-splitter, not a dispersing prism: it peels off three clean bands, it does not smear a spectrum.)

This strategy spends no scene axis, so it makes none of the constancy or smoothness assumptions of the other two: simultaneous sampling on three full grids without CFA demosaicking. Instead you pay hardware complexity: three sensors registered to sub-pixel accuracy behind a bulky, expensive prism. Bulk, optical tolerances, and cost limit its use. Adding bands increases the complexity.

The hardware-splitting idea predates its dichroic form. Before interference coatings existed, P. D. Brewster’s historical beam splitter divided a movie camera's beam with a polished metal disc drilled full of holes (Figure 4.6.8⛶ fullscreen): light passing through the holes went straight to one color film gate, while light striking the solid metal between them reflected off to a second. It is the cheese-grater ancestor of the 3-CCD prism, the same bargain of spending hardware rather than a scene axis so that each color gets its own sensor. It was cruder in two ways: the fixed metal-to-hole ratio sets a single reflect/transmit split rather than a wavelength-selective peel, and because the disc splits by geometry and not by color, each path still needs its own color filter. So it lacks the dichroic prism's wavelength-selective separation, but the core move is identical.

A dichroic coating separates bands through interference. The answer is the same interference that colors a soap bubble (Figure 2.1.33), run deliberately and many times over. The coating is a quarter-wave stack: alternating layers of a high- and a low-index transparent material, each with optical thickness $n_jd_j=\lambda_0/4$ at normal incidence in the simple quarter-wave design. Every interface reflects a little, and at $\lambda_0$ those weak reflections all come back in phase and add into a near-perfect mirror; outside the reflection band the coating transmits more light, with the detailed spectrum set by its complete layer design. In an ideal lossless coating, reflectance plus transmittance is one. Real stacks also have absorption, scattering, angular dependence, and polarization dependence (Figure 4.6.9⛶ fullscreen).

Multiplexing per pixel by dispersion: color routers and nano-prisms🔗⧉

A color router directs wavelength bands toward different photodiodes instead of rejecting unwanted bands with absorbing filters. Demonstrations range from dielectric splitters Nishiwaki et al. 2013 to inverse-designed structures Zou et al. 2022. The target is better photon utilization, but reported efficiency depends on wavelength, incidence angle, polarization, aperture, and the definition of useful detector area. One-third transmission is a teaching approximation for idealized equal-width RGB filters, not a universal CFA ceiling.

Routing to neighboring photodiodes still multiplexes color in space. The spatial response must be included in reconstruction; spreading light is not automatically a correctly designed anti-aliasing filter. Fabrication tolerances and off-axis response also matter. Commercial microlens or nanostructure enhancements that retain color filters should be distinguished from filter-free research routers.

The comparison with a dichroic prism is useful, provided we keep track of the image coordinates. A prism can send complete registered images to separate sensors. A small router sends different wavelengths toward different sites within a local footprint. The latter can avoid absorbing some of the photons that a dye would reject, but neighboring detectors now receive contributions shaped by the router's wavelength-dependent spatial response. The reconstruction must know both which wavelengths and which scene locations contributed. Improving light collection and preserving spatial detail are therefore coupled design tasks, not two independent benefits of replacing a filter.

Multiplexing per pixel by tunable absorbers: quantum dots🔗⧉

Colloidal quantum dots offer size- and material-dependent bandgaps and can form photosensitive layers above a readout circuit. They are useful research and device materials for spectral ranges and integration schemes that differ from silicon. Color-selective layers still need multiple independent measurements, whether distributed spatially or stacked.

A tunable bandgap does not imply an arbitrarily narrow absorption spectrum. Quantum dots can absorb broadly at energies above the band edge, even when their fluorescence is narrow. Nor does replacing a dye with another absorber automatically recover rejected photons. Evaluate absorption, charge extraction, dark current, stability, and cross-channel separation for the proposed architecture.

Multiplexing in depth: stacked photodiodes🔗⧉

Stacked photodiodes use silicon’s wavelength-dependent absorption depth. Short wavelengths are absorbed near the surface; longer wavelengths penetrate farther. The layer signals overlap spectrally, so they require a calibrated color transform rather than being direct red, green, and blue measurements. Foveon designs use this principle; some variants sample deeper layers at lower spatial resolution and still require reconstruction. Noise after color correction depends on the measured layer sensitivities, collection efficiency, and covariance.

Few production cameras have ever used a Foveon sensor; the Sigma dp series is the best known (Figure 4.6.16⛶ fullscreen), sold on exactly the trade above: unusually crisp, demosaicking-free luminance detail in exchange for noisier color, device-dependent sensitivity and readout limits.

For a simplified collection interval from depths $z_1$ to $z_2$, Beer–Lambert absorption gives the fraction $e^{-\alpha(\lambda)z_1}-e^{-\alpha(\lambda)z_2}$, where $\alpha$ is silicon’s absorption coefficient. This illustrates overlapping depth responses (Figure 4.6.17⛶ fullscreen). Real charge collection also depends on device structure and carrier transport. A broad overlap can make some color distinctions noisy; quantify that through the calibrated transform rather than an invented “unmixing” matrix.

Depth multiplexing is not a silicon invention; color film got there first, chemically. The dominant color-film design, the integral tripack of Kodachrome, Ektachrome, and typical C-41 color negatives, stacks three color-sensitive layer groups, often containing several emulsions, on one base: a blue-sensitive layer on top, then a green-sensitive layer, then a red-sensitive layer, with a thin yellow filter layer just below the top one to stop stray blue (to which all silver halides respond) from leaking into the lower layers. White light enters the top and is caught band-by-band with depth, a similar stacked layout, though spectral sensitization in film differs from absorption-depth separation in silicon, and after development each layer forms its own complementary dye so the stack yields full color at every point with no spatial mosaic (Figure 4.6.18⛶ fullscreen). It is the same trade the Foveon makes and the same one the Autochrome does not: color multiplexed in depth (stacked, full-resolution, no demosaicking) rather than across space (a mosaic that must be interpolated). Silicon's Foveon is the integral tripack rebuilt in a semiconductor, absorption depth standing in for the sensitized layers.

The depth layers are not three bins labeled blue, green, and red. Absorption is probabilistic: at a given wavelength, some photons are absorbed near the surface and others penetrate farther. A shallow collection region therefore receives longer-wavelength contributions as well as short-wavelength ones; deeper regions measure the surviving mixture. The equation above computes the difference between the fractions that reach the top and bottom of an idealized collection interval. This is why three layers need a calibration rather than merely being renamed RGB. When a desired color distinction changes the layer signals only slightly, recovering it can require a large difference of noisy measurements, connecting the sensor architecture to the matrix-noise calculation in Chapter 4.4, RGB color spaces.

Hybrid strategies🔗⧉

Real cameras often combine strategies, and the most useful hybrid is Bayer plus pixel-shift (Figure 4.6.19⛶ fullscreen). A sensor-shift actuator moves through a specified sequence of one-photosite offsets, so that over four exposures every output location is sampled through a red, a blue, and both green filters in turn; merge the four and you get true R, G, B at every pixel with no demosaicking guesswork (full color resolution and additional samples for noise reduction) at the cost of needing a static scene on a tripod. (It is the spatial Bayer trick borrowing the temporal axis to fill its own holes.) Subpixel offsets can support super-resolution if the optics retain the detail and registration is accurate. Motion and occlusion require detection or fallback processing; more frames also mean more capture time. Video pipelines hybridise the other way, leaning on motion across frames; astrophotography filter wheels are temporal multiplexing with cooled sensors.

Beyond trichromatic capture: full spectrum and multispectral🔗⧉

Multispectral instruments acquire more than three bands; hyperspectral systems sample many closely spaced bands. Filter wheels, scanned spectrometers, filter mosaics, and snapshot spectral encodings trade spectral detail against spatial sampling, time, and noise. A spectral cube is still a finite set of measurements with a calibrated spectral response, not an exact continuous spectrum.

Lippmann photography records spectral information through interference along the depth of an emulsion Lippmann 1891 Lippmann color photography (Alternative Photography) Bjelkhagen, "Lippmann photography". Incident light and its reflection from a backing mirror form depth-dependent intensity fringes. For vacuum wavelength $\lambda$, refractive index $n$, and internal angle $\theta$, adjacent intensity maxima are separated by

After development, the layered structure selectively reflects wavelengths under suitable viewing illumination. Finite thickness limits spectral resolution; the mirror’s phase response, emulsion response, processing shrinkage, and viewing angle affect reproduction. It preserves more spectral information than an RGB photograph, but not a perfect, unlimited record of the original spectrum. Modern recovery methods model these distortions Baechler et al. 2021.

For a monochromatic exposure, the incident and reflected waves establish a repeated intensity pattern through the depth of the emulsion. Development records that pattern as a layered optical structure. When white light later illuminates it, reflections from successive layers reinforce one another near the recorded wavelength under the appropriate viewing geometry. A more complicated incident spectrum leaves a more complicated depth pattern. The layer spacing carries spectral information even though the emulsion is not divided into conventional red, green, and blue channels. Lippmann's method is therefore a useful counterexample to the assumption that a color photograph must store three numbers: it records another kind of optical measurement, whose reconstruction still has finite bandwidth and physical imperfections.

Spectral sensitivity: matching the eye, catching photons, and white balance🔗⧉

With photon spectral exposure $q(\lambda)$ and effective channel detection probability $\eta_i(\lambda)$, the expected signal electrons are

The response includes filters, optics, and quantum efficiency under the stated convention. Sampled implementation needs wavelength-bin weights. Shot noise is determined by detected electrons, while read noise is added by the readout; the full model is in Chapter 2.8, Noise, signal-to-noise ratio and dynamic range. A broad response alone is not a color measurement: distinct response shapes supply the information that separates spectra.

The exact shape of a color sensor's three spectral-sensitivity curves (the $r(\lambda)$ of the red, green, and blue channels, built earlier by the color-filter array on the photosites of Chapter 2.7, Sensors: photosites, CCD vs CMOS) is one of the camera's most constrained choices, because it is pulled in three directions at once that cannot all be satisfied.

• Fidelity to human vision. For a camera to record the color a human would see, its three curves should be a fixed linear combination of the eye's cone fundamentals (equivalently, of the CIE color-matching functions), the Luther–Ives condition. A sensor that met it could be mapped to human color by a single $3\times3$ matrix with no error. Practical camera sensitivities generally only approximate it, because silicon-plus-dye curves are simply the wrong shape, and the penalty is a metameric error no color matrix can remove: two spectra that look identical to the eye read as different RGB, and two that look different can read the same. The color-correction matrix in Chapter 8.10, Auto-exposure and auto white balance is a least-squares compromise: right on average, wrong on the hard cases (saturated reds, narrow-band LEDs, foliage).

• Quantum efficiency and noise. Broad transmission bands can increase total photon collection; this is distinct from quantum efficiency at a specified wavelength: the wider each filter, the more photons reach the well and the higher the SNR. Strong overlap can make color separation poorly conditioned, because separating similar channel responses can require large signed coefficients that amplify measurement noise (compare the distinct physiological opponent processing in Chapter 3.2, Perceptual color and trichromatic vision). Narrow, well-separated filters give clean color but throw photons away and read noisier. The green-doubled Bayer is one point on this curve: spend photons and resolution where the eye cares most (luminance), economize on chroma.

• White balance across illuminants. Because each channel integrates illuminant × reflectance × sensitivity, changing the light changes the raw RGB of a fixed object. Undoing that, white balance often approximates the change with per-channel gains. Matching cone sensitivities does not make this diagonal relighting model exact; a response set that matches human colorimetry well need not be the best basis for diagonal adaptation. Good spectral design makes white balance more reliable across illuminants (→ Chapter 8.10, Auto-exposure and auto white balance).

These three pull against each other: the curves that best match the eye are broad and overlapping (bad for separation), the curves that separate color cleanly are narrow (bad for QE and for matching the eye), and keeping white balance stable across odd illuminants wants yet another shape. Every color sensor is a negotiated settlement among them (Figure 4.6.24⛶ fullscreen).

RAW files: before the cooking🔗⧉

Raw storage retains data from an earlier stage of the camera pipeline, leaving more reconstruction and rendering choices to the developer. The file may contain a CFA mosaic, demosaicked channels, or partly processed measurements. Raw capture offers latitude, but does not guarantee unmodified or losslessly stored sensor data.

To obtain scene-linear working values, decode the stored representation, apply any linearization table, subtract calibrated black levels, and use the metadata-defined scale. A linear DNG can already contain reconstructed channels. Check processing tags and profiles rather than assuming the extension establishes the numerical domain.

Here is the subtlety, stated plainly: "RAW" is not always purely raw. The name suggests an untouched sensor dump, but in practice most cameras pre-cook the data to some degree: subtracting the sensor's black level, correcting defective and hot pixels, sometimes applying lens-shading or vignetting correction or even mild denoising. Some "RAW" formats are themselves lossy-compressed or non-linearly encoded to save space. So RAW is best thought of as a spectrum of how cooked the data is, from a near-literal readout to something already partly processed, not a single guarantee of pristineness.

There is also a practical headache: RAW is largely proprietary. Nearly every manufacturer ships its own format and extension: Canon's (CR3 files), Nikon's (NEF files), Sony's (ARW files), and so on, each subtly different. That creates software and archival problems, since a format only one company supports may not be readable in thirty years. Adobe's Digital Negative (DNG) format is an open, documented, standardized RAW container meant to fix exactly this (its specification Adobe DNG specification is the reference here); many photographers convert their proprietary RAWs to DNG for safekeeping, and some cameras shoot DNG directly.

import rawpy
with rawpy.imread("photo.dng") as raw:
    bayer = raw.raw_image.copy()  # retain samples after closing raw
    pattern = raw.raw_pattern
    pattern = None if pattern is None else pattern.copy()
    rgb   = raw.postprocess()      # or let LibRaw develop it to RGB

Demosaicking and other aspects of imaging🔗⧉

Demosaicking is one part of an imaging system. The optics and the sensor decide what the mosaic records, and the stages of the camera pipeline around it (Part 8, Basic image processing and ISP) decide what it is handed and what happens to its output. Each method in this chapter and in Chapter 15.2, Advanced traditional demosaicking assumes something about its input, and these other parts decide whether the assumption holds.

The anti-aliasing filter. Many cameras place an optical low-pass filter in front of the sensor. It blurs the image slightly before sampling, so that less fine detail aliases into color moiré, the failure no demosaicker can fully undo because the mosaic cannot tell it from real color. Many recent cameras have weakened or dropped it and leave moiré to the demosaicker. Chapter 15.3, Mosaics and aliasing explains what the filter can and cannot remove, and why it has been disappearing.

Grouped layouts: Quad and Nona. Phone sensors that group same-color photosites in 2 × 2 (Quad-Bayer) or 3 × 3 (Nona-Bayer) blocks change what a demosaicker receives. In low light the sensor bins each block and hands over an ordinary Bayer mosaic at a quarter or a ninth of the pixel count, and the methods of this chapter apply unchanged. At full resolution the same color is repeated over neighboring sites, so red, for example, is missing from larger regions than in a Bayer mosaic, and the nearest red sample can be three photosites away. A Bayer demosaicker applied directly to that layout uses the wrong colors at most sites; the usual route is to remosaic the grouped layout into an estimated Bayer mosaic first, which is itself an interpolation, then demosaick, or to demosaic the grouped layout directly (Chapter 15.10, Quad-Bayer and Nona-Bayer processing).

Color routers. A nano-prism or metasurface color router keeps the Bayer layout but steers each band of light toward the photosite that measures it instead of absorbing it (Chapter 4.6, Sensing color: multiplexing strategies). Each photosite then collects light from a wider area than its own, which acts as a mild anti-aliasing filter (Chapter 15.3, Mosaics and aliasing), and part of each band lands on neighbors of the wrong color, which broadens and mixes the channels' spectral responses. The demosaicking algorithms still apply, but the color correction that follows has to undo more mixing.

Black level. The sensor adds an offset to every sample, and it has to be subtracted first. Color differences and ratios are meaningful only on values proportional to light: with the offset left in, a gray edge no longer has a constant $R-G$ across it, and the ratio model of the "difference or ratio?" sidebar is wrong everywhere.

Defects and lens shading. A hot or dead photosite is one wrong sample, and interpolation copies it into its neighbors' missing colors, so a single bad value becomes a colored speck several pixels across; repair it on the mosaic. Lens shading darkens the corners by a different amount in each channel, which demosaicking would read as color. Both are covered in Chapter 15.7, Neighboring problems in the raw stage.

White balance. The color-difference methods work best when a neutral surface has equal values in all three channels, because then the difference across a gray edge is zero on both sides and interpolating it is exact. Raw values under most lights are far from equal for a neutral, so many pipelines apply the white-balance gains to the mosaic before demosaicking (Chapter 8.10, Auto-exposure and auto white balance). The gains then scale clipped samples too: a highlight that clipped in green is no longer at a common ceiling once red and blue are multiplied, which is where pink highlights come from, so saturated samples need to be marked before the gains are applied.

Chromatic aberration. Lateral chromatic aberration shifts the color channels relative to one another toward the edges of the frame. The cross-channel assumption then fails exactly at edges, where it is needed, and a demosaicker reads the misregistration as a colored edge. Correcting it before or during demosaicking avoids fringes that are much harder to remove afterward (Chapter 22.2, Aberrations correction).

Noise. Interpolation mixes neighboring samples, so independent sensor noise becomes spatially correlated color noise, which is harder for a denoiser to recognize. Noise also flips the edge-direction decisions of this chapter. Denoising the mosaic first, or demosaicking and denoising together, avoids both (Chapter 15.8, Joint demosaicking and denoising).

After demosaicking. The color matrix, tone curve and sharpening all act on the reconstructed image, and sharpening in particular amplifies zippering and maze patterns along with real detail (Chapter 8.8, Sharpening). A demosaicker that leaves small artifacts can look acceptable until the rest of the pipeline has run, so judge it on the finished rendering as well as on its direct output.

The practical rule is to know which of these steps have been applied to the data a demosaicker receives, in what units, and to tell the algorithm.

And all of it should happen on linear values. Black-level subtraction, defect repair, lens-shading and white-balance gains, and demosaicking itself operate on numbers proportional to light, and the image becomes nonlinear only afterward, when the tone curve and the gamma encoding prepare it for display. The reasons are the ones this chapter keeps meeting. Interpolation averages, and an average of gamma-encoded values is not the encoding of the average light, so a demosaicker run on encoded values darkens fine detail and shifts colors at edges (Chapter 6.5, Resampling and upsampling). Color differences and ratios describe the scene only when the values are proportional to light. And the noise model that joint methods rely on is simple only in linear units, before a curve stretches the shadows and compresses the highlights.

Advanced traditional demosaicking🔗⧉

The methods below, published between the late 1980s and the mid-2000s, combine the two ideas of Chapter 15.1, Basic traditional demosaicking, edge direction and color differences, in different proportions, and hand-designed methods of this family are still what most cameras and phones run in their image signal processors, where a demosaicker must run on every preview and video frame, fast, small and predictable. Learned demosaicking has shipped mainly in desktop raw converters, such as Adobe's Enhance Details and DxO's DeepPRIME, which also denoises (Chapter 15.6, Learned demosaicking). Some phone pipelines avoid single-frame demosaicking altogether by merging a burst of raw frames directly onto a full-color grid (Chapter 14.2, Multi-frame super-resolution). Figure 15.2.1⛶ fullscreen runs ten of them side by side on the same mosaic, including the residual interpolation and ARI of this chapter, so each method's assumption can be seen where it fails.

Nearest neighbor copies each missing value from a nearby sample of the right color, with no arithmetic. The name is loose, because most sites have several equally near samples of a given color: a red site has four green neighbors at the same distance, and four blue ones on the diagonals. An implementation breaks the tie with a fixed rule, usually by working tile by tile. In the lab's version, every pixel of a 2 × 2 tile takes that tile's one red and one blue, a green site keeps its own green, and the other two sites take the same one of the tile's greens. That makes it a close relative of the downsampling of Downsampling and bilinear interpolation, and how they fail: downsampling turns each tile into one pixel, and nearest neighbor is almost that same small image blown back up to full size by copying each value into the four pixels of its tile. The only differences are that the measured greens stay where they were and nothing is averaged. Nearest neighbor therefore keeps the pixel count but adds almost no detail over downsampling, and it keeps downsampling's flaw: each color is attributed to a place up to a pixel away from where it was measured, so every edge becomes a colored staircase, the strongest zipper of any method here. It costs nothing, which is why it survives in live previews and thumbnails. Bilinear interpolation (Downsampling and bilinear interpolation, and how they fail) averages instead of copying and fixes the misregistration in smooth regions, not at edges.

Figure 15.2.2⛶ fullscreen works through every method of this section on a small mosaic, one step per stage, starting with nearest neighbor.

Smooth hue transition Cok 1987 (smooth hue transition), one of the first methods to use one channel to reconstruct another, interpolates green first and then interpolates hue rather than red and blue: the ratio $R/G$ at the red sites, spread to its neighbors and multiplied back by green. It assumes that hue changes slowly across an object even where brightness does not. The choice of hue coordinate matters, and the lab lets you switch it: a ratio becomes unstable where green is near zero, a difference $R-G$ does not, and a log ratio sits in between. The difference form is the recipe of Interpolate differences, not channels.

Its three steps are the smooth-hue stages of Figure 15.2.2⛶ fullscreen.

Freeman's method Freeman 1988 (median demosaicking) demosaicks with bilinear interpolation, then cleans up the hue: it applies a 3 × 3 median to the difference channels $R-G$ and $B-G$ and adds green back. Within an object the color difference is nearly constant, so a false-color pixel is an isolated outlier in a flat channel, which is exactly what a median removes. A 5 × 5 median often does a better job, because false color on fine texture tends to come in clusters wider than a 3 × 3 window can outvote, though a larger median also starts to carry color across object boundaries. The median does nothing for zippering in the green channel, so although the method is described on top of bilinear, it is mostly used as a clean-up step on top of a better demosaicker. Figure 15.2.1⛶ fullscreen offers both bases: on bilinear, the median removes most of the fringes and leaves the zippering; on the chapter's edge-based demosaicking, the green is already good and the median only has the remaining false color to remove. The same pass survives as the final clean-up of many later methods, AHD among them. (Freeman's patent differs in its details: it describes two colors alternating along a line, interpolates each linearly, and applies a one-dimensional median to their difference, five pixels wide in its example, with each pixel keeping its own measured value. The two-dimensional Bayer version above is the one usually implemented.)

Figure 15.2.2⛶ fullscreen shows the three steps on the grid.

Gradient-corrected linear interpolation Malvar et al. 2004, by Malvar, He and Cutler, keeps a fixed linear filter but adds the Laplacian of the channel measured at the center pixel, scaled by a gain, to the bilinear estimate of each missing channel. A red pixel's curvature, for example, corrects the green estimated there, on the assumption that the channels vary together. The result is a set of fixed 5 × 5 filters with no decisions, so it is fast, easy to run on a GPU, and free of the unstable direction choices that make mazes; it is, for example, what MATLAB's demosaic function implements. Because it does not adapt, it still blurs across edges where an adaptive method would not.

The same correction, pixel by pixel, is in Figure 15.2.2⛶ fullscreen.

Hamilton–Adams Hamilton & Adams 1997, met in Interpolate along the edge, not across it, combines both ideas of the basic chapter, and it does so in three passes over the mosaic, each using the channels already filled in.

1. Green at the red and blue sites. Measure a horizontal and a vertical activity at the site, each the sum of a green gradient across the site and the second difference of the center color along the same line. Interpolate green along the less active direction, as the average of its two green neighbors plus a quarter of the center color's second difference, and use all four neighbors when the two activities are equal. This is the pseudocode of Interpolate along the edge, not across it.
2. Red and blue at the green sites. Now that green exists everywhere, a missing red at a green site is the average of its two red neighbors, horizontal or vertical depending on the row, corrected by half of green's second difference along that line. It is color-difference interpolation in disguise: the result equals the site's green plus the average of the two neighbors' $R-G$, up to the correction.
3. Red at the blue sites, and blue at the red sites. The four nearest samples of the missing color sit on the diagonals, so the method chooses between the two diagonals with the same kind of test, a gradient of the missing color along the diagonal plus green's second difference along it, and interpolates along the calmer one with the same half-weight correction.

# Hamilton-Adams, steps 2 and 3 (step 1, green, is in the edge-directed section)
# step 2: red at a green site in a red row (blue rows: use the vertical pair)
R[x,y] = (R[x-1,y] + R[x+1,y])/2 + (2*G[x,y] - G[x-1,y] - G[x+1,y])/2

# step 3: red at a blue site, choosing a diagonal
dN = abs(R[x-1,y-1] - R[x+1,y+1]) + abs(2*G[x,y] - G[x-1,y-1] - G[x+1,y+1])   # NW-SE
dP = abs(R[x+1,y-1] - R[x-1,y+1]) + abs(2*G[x,y] - G[x+1,y-1] - G[x-1,y+1])   # NE-SW
if   dN < dP: R[x,y] = (R[x-1,y-1] + R[x+1,y+1])/2 + (2*G[x,y] - G[x-1,y-1] - G[x+1,y+1])/2
elif dP < dN: R[x,y] = (R[x+1,y-1] + R[x-1,y+1])/2 + (2*G[x,y] - G[x+1,y-1] - G[x-1,y+1])/2
else:         R[x,y] = (R[x-1,y-1] + R[x+1,y+1] + R[x+1,y-1] + R[x-1,y+1])/4 \
                       + (4*G[x,y] - G[x-1,y-1] - G[x+1,y+1] - G[x+1,y-1] - G[x-1,y+1])/4
# blue at a red site: the same, with R and B exchanged

The corrections are what separate it from bilinear. Each one assumes that the missing channel curves the same way as the channel that was measured or already reconstructed at the site, so a sharp edge in green is copied into red and blue at the same place instead of being smeared into a fringe. The direction tests are what separate it from Malvar–He–Cutler, which applies the same kind of correction everywhere, in every direction. Hamilton–Adams costs a small, fixed amount of work per pixel, and its green step is the starting point of many later methods, including the residual interpolation of the next chapter; its failures are those of any two-way choice: diagonal texture, and fine patterns where both directions look equally active.

Figure 15.2.2⛶ fullscreen runs the three passes on the grid; the grating near 45 degrees shows the failure.

Variable number of gradients Chang, Cheung & Pan 1999 (VNG), VNG, by Chang, Cheung and Pan, replaces the two-way choice with a vote among eight directions, and lets the image decide how many of them to trust. At each pixel it works in the 5 × 5 neighborhood around the site, in four steps.

1. Eight gradients. For each compass direction, north, northeast, east and so on, sum the absolute differences between pairs of same-color samples that lie along that direction on that side of the pixel. Pairing same-color samples keeps the gradient free of the differences between channels. A direction that points into a smooth region gets a small gradient; one that crosses an edge gets a large one.
2. A threshold. From the smallest gradient $g_{\min}$ and the largest $g_{\max}$, set $T = 1.5\,g_{\min} + 0.5\,(g_{\max}-g_{\min})$. The threshold is relative: it asks which directions are nearly as smooth as the smoothest one here, not which are below a fixed number, so the same rule works in shadows and highlights.
3. A vote. Keep every direction whose gradient is below $T$. For each kept direction, average the samples of each color that lie on that side of the pixel, and sum these averages over the kept directions, one sum per color.
4. A color difference. Estimate each missing color from the center sample and the difference of the sums: at a red site, for instance, $\hat G = R + (S_G - S_R)/n$, where $S_G$ and $S_R$ are the sums for green and red over the $n$ kept directions. This is the color-difference idea of Interpolate differences, not channels, averaged over the directions the vote allowed.

# VNG at one pixel, in outline (per-site sample weights omitted)
g = {d: gradient(d) for d in [N, NE, E, SE, S, SW, W, NW]}       # same-color pairs along d
T = 1.5*min(g.values()) + 0.5*(max(g.values()) - min(g.values()))
keep = [d for d in g if g[d] <= T]
S = {c: sum(average_on_side(c, d) for d in keep) for c in [R, G, B]}  # one sum per color
for c in missing_colors:
    value[c] = center_value + (S[c] - S[center_color]) / len(keep)

The number of directions that pass is what gives the method its name and its behavior. In a flat region all eight pass and it averages widely, which suppresses noise; along a straight edge only the two directions parallel to it pass, and it averages along the edge like Hamilton–Adams; at a corner or a line end it can keep three or four, which a two-way choice cannot express (Figure 15.2.4⛶ fullscreen). The cost is about eight gradients per pixel over a 5 × 5 window, several times Hamilton–Adams, and the relative threshold can pass too many directions in fine, low-contrast texture, where everything looks equally smooth. VNG was long one of the modes of the dcraw raw decoder.

Figure 15.2.2⛶ fullscreen runs VNG on every pixel of the grid.

Two further refinements are common. Iteration: once green and the color differences are in, re-estimate the differences against the interpolated green and repeat. Alternating projections Gunturk, Altunbasak & Mersereau 2002 (alternating projections) formalize this as projecting in turn onto two sets, the images that agree with the measured samples and the images whose red and blue have the same fine detail as green. And directional estimation with explicit noise models: Zhang and Wu's directional linear minimum mean-square-error method Zhang & Wu 2005 (DLMMSE) fuses the horizontal and vertical estimates with weights set by how noisy each one is, rather than choosing one.

Across the eight panes, three things change from method to method. The assumption about images: smooth channels, smooth hue, correlated channels, or edges along one of a few directions. The decision made at each pixel: none for bilinear and Malvar–He–Cutler, a direction for Hamilton–Adams, VNG and AHD (#Adaptive homogeneity-directed demosaicking (AHD), next). And the cost, from nothing for nearest neighbor to two full reconstructions and a color-space conversion for AHD. The residual-interpolation methods (#Residual interpolation and ARI) take the next step by replacing the color difference with the residual of a guided fit.

For reproducible comparisons, state the implementation and demosaicking mode. LibRaw and dcraw expose several algorithms, but their defaults are not interchangeable with the Malvar–He–Cutler method. Compare against a strong classical baseline such as adaptive residual interpolation Monno et al. 2015 (adaptive residual interpolation) as well as a simple linear method.

The sections that follow take up three families that go further: AHD, which builds whole reconstructions in two directions and keeps the more homogeneous one; residual interpolation and ARI, which interpolate what a guided prediction leaves over; and frequency-domain demultiplexing, which treats the mosaic as a modulated signal. The chapter ends with the situations that defeat all of them.

Adaptive homogeneity-directed demosaicking (AHD)🔗⧉

If you want to implement one classical demosaicker that is simple and works well, this is probably your best bet, followed by Freeman's median of the color differences (earlier in this chapter) to remove the false color that remains. Every method so far decides the interpolation direction before interpolating, from gradients in the mosaic. Adaptive homogeneity-directed demosaicking Hirakawa & Parks 2005 (AHD), by Hirakawa and Parks, decides afterward. It reconstructs the whole image twice, once interpolating only horizontally and once only vertically, and at each pixel keeps whichever reconstruction looks more plausible there. The criterion is the key idea: interpolating in the wrong direction averages across an edge and creates colors that exist on neither side of it, so the wrong candidate is less homogeneous, locally, than the right one.

1. Two candidates. Build a full image with every interpolation along rows: green with a Hamilton–Adams-style correction, then red and blue by interpolating color differences. Build a second with every interpolation along columns.
2. A perceptual space. Convert both candidates to CIELAB, so that lightness and color can be compared separately and a distance means roughly what a viewer would notice (Chapter 4.5, Advanced color: opponents, perceptually-uniform, and preferred).
3. Homogeneity. For each pixel of each candidate, count the neighbors in a small window that are close to it in both lightness and color, within thresholds $\varepsilon_L$ and $\varepsilon_C$. The thresholds are set at every pixel from the differences to its nearest neighbors along each candidate's own direction, so the test adapts to local contrast rather than using one number for shadows and highlights.
4. The choice. Sum each candidate's counts over a 3 × 3 window and keep, at each pixel, the candidate with the larger sum; average the two on a tie.
5. Clean-up. Finish with a few passes of a median over the color differences $R-G$ and $B-G$, Freeman's clean-up.

# AHD, in outline
Ih, Iv = demosaic_along_rows(mosaic), demosaic_along_columns(mosaic)
Lh, Lv = to_cielab(Ih), to_cielab(Iv)
eps_L, eps_C = adaptive_thresholds(Lh, Lv)             # per pixel, from nearest-neighbor differences
Hh = count_close_neighbors(Lh, eps_L, eps_C)           # lightness and color both within threshold
Hv = count_close_neighbors(Lv, eps_L, eps_C)
Hh, Hv = box_sum_3x3(Hh), box_sum_3x3(Hv)
out = where(Hh > Hv, Ih, where(Hv > Hh, Iv, (Ih + Iv) / 2))
out = median_color_differences(out, passes=3)

Figure 15.2.5⛶ fullscreen shows the homogeneity test at one pixel.

The figure below runs it on a real facade, where the cornice's horizontal lines choose the horizontal candidate and the window columns the vertical one (Figure 15.2.6⛶ fullscreen), and Figure 15.2.2⛶ fullscreen makes the same choice cell by cell on the chapter's small inputs.

Figure 15.2.2⛶ fullscreen steps through the same algorithm on the small inputs.

AHD costs two full reconstructions, two color-space conversions and the counting, several times Hamilton–Adams. It was the default of the dcraw raw decoder and of many converters built on it for years, and it is still a strong classical baseline. Its limits are the other methods' limits. It still chooses between two directions, so diagonal texture gives both candidates errors and the choice cannot fix them. In fine texture near the sampling limit both candidates are inhomogeneous and the choice becomes erratic, which is where mazes come from. And noise lowers the homogeneity of both candidates everywhere, so the choice degrades as the noise rises, one reason to denoise first or jointly (Chapter 15.8, Joint demosaicking and denoising).

Residual interpolation and ARI🔗⧉

The best-performing classical demosaickers, and the ones to compare against today, come from the residual-interpolation family, and adaptive residual interpolation (ARI) is its strongest member. It appeared a decade after AHD, outperforms it clearly in published comparisons on the standard test sets, and needs no training. It is also built directly on the guided filter of Chapter 10.6, Guided image filtering, so it is a payoff of that chapter rather than a new tool. It takes more work to implement than AHD, which remains the simpler choice when that matters.

The idea changes what gets interpolated. Every classical method interpolates something, and each step of the classical line chose a flatter quantity. Bilinear interpolates the channel itself and fails at edges. Hamilton–Adams and its successors interpolate the color difference $G-R$, which is smooth wherever the two channels differ by an offset, and fails where that assumption does: at a saturated color boundary, at a specular highlight, wherever the ratio between the channels changes rather than their offset. Residual interpolation Kiku et al. 2013 (residual interpolation), by Kiku, Monno, Tanaka and Okutomi, goes one step further. Rather than hope that some quantity is flat, it constructs one: it fits a local prediction of the sparse channel from the dense one and interpolates only what the prediction gets wrong.

Take red, sampled on a quarter of the pixels, and green, already reconstructed everywhere because it is sampled twice as densely. In a small window, model red as a linear function of green,

with $a$ and $b$ estimated from the red samples that exist in that window. This is a guided filter with green as the guide He, Sun & Tang 2013 (guided filter), restricted to the measured sites. In a window $\Omega$, with $\mathcal R$ the set of sites where red was measured, solve

where $\epsilon>0$ stabilizes the slope. Unmeasured sites must not enter the fit as zeros; a window with no red sample has to grow or fall back to a simpler estimate; and since every pixel lies in many windows, the overlapping predictions are averaged, which is the guided filter's own rule. The tentative estimate $\tilde R$ is a full-resolution red built entirely out of green's structure, and it follows a change of contrast between the channels that a difference cannot. The residual is the part of red that green cannot explain,

and the reconstruction interpolates it, by any simple method, and adds it back:

The prediction carries the shared detail, and the residual corrects its errors where measurements constrain them. When the local model explains the structure the channels share, the residual is smaller and flatter than the color difference, and the same cheap interpolator does much less damage to it. That flatness is an assumption to test, not a property the fit enforces: a small residual at the measured sites does not prove that the residual between them is small or smooth, and a misleading guide, a sharp color boundary or a poorly conditioned fit can transfer the wrong structure into the result.

The full algorithm applies the same idea twice, first to green and then to red and blue.

1. Green first. Estimate green along rows and along columns separately, each time guiding by the measured red or blue along that line and interpolating the residual along it, then combine the two directional estimates with weights that favor the direction whose color differences vary less locally.
2. Tentative red. With the full green as the guide, fit a local linear model $\tilde R = aG + b$ in each window, using only the sites where red was measured, and evaluate it everywhere.
3. Residual. At the measured red sites, form $R-\tilde R$.
4. Interpolate and add back. Interpolate the residual bilinearly to every site and add it to $\tilde R$. Blue is the same with blue's samples.

On a red flower against green leaves, the residual is close to zero where the color difference is large, and residual interpolation scores 3.3 dB above AHD (Figure 15.2.8⛶ fullscreen).

Minimized-Laplacian residual interpolation, MLRI Kiku et al. 2014 (minimized-Laplacian residual interpolation), changes only how $a$ and $b$ are fitted: instead of least squares on the values, it minimizes the Laplacian energy of the residual, which is what the final bilinear interpolation needs to be small. It changes how the regression coefficients are estimated, not which channel serves as the guide. ARI Monno et al. 2015 (adaptive residual interpolation), by Monno, Kiku, Tanaka and Okutomi, iterates both, using each output as the next guide, and at each pixel selects the variant and the number of iterations whose residual is smoothest, judged by the local fit. That is the direction-selection idea of Hamilton–Adams operating one level up: not which direction to interpolate in, but which method to trust here. A later version extends it to multispectral filter arrays Monno et al. 2017 (ARI, multispectral). The authors' reference code has been used widely enough in later work that comparisons against it can be trusted.

# Residual interpolation for red, green already reconstructed (blue is the same)
def residual_interpolation(G, R_mosaic, red_mask, radius, eps):
    a, b = masked_guided_fit(guide=G, target=R_mosaic, mask=red_mask, radius=radius, eps=eps)
    R_tent = box_mean(a, radius) * G + box_mean(b, radius)     # guided filter output
    residual = where(red_mask, R_mosaic - R_tent, 0)
    return R_tent + bilinear_from_samples(residual, red_mask)

# MLRI: the same, with a and b fitted to minimize the Laplacian of the residual
# ARI, in outline: at each pixel keep the (method, iteration) whose residual is smoothest
estimates = []
guide = G
for k in range(K):
    for method in [RI, MLRI]:
        R_k = method(guide, R_mosaic, red_mask)
        estimates.append((R_k, laplacian_energy(R_mosaic - R_k, red_mask)))
    guide = R_k
R = per_pixel_argmin(estimates)          # smoothest residual wins, in a local window

Figure 15.2.2⛶ fullscreen steps through residual interpolation cell by cell on the chapter's small inputs, from the directional green to ARI's per-pixel choice, and the demosaicking lab at the start of this chapter (Figure 15.2.1⛶ fullscreen) runs the same code on photographs.

The limits are the guided filter's. A local linear model cannot follow channels that are related nonlinearly, at clipping or under strong chromatic aberration, and fitting $a$ and $b$ from a few noisy samples is unstable, so noise should be removed first, accounted for within the fit, or handled by joint reconstruction (Chapter 15.8, Joint demosaicking and denoising).

This family, not bilinear interpolation, is the baseline a learned demosaicker should be measured against. It embodies three separate decisions: predict one channel from a guide, interpolate what the prediction missed, and choose locally between useful variants. A learned model can change any of the three, and a comparison with bilinear alone leaves those established improvements unaccounted for, so it cannot tell which part of a network's gain required learning. Compare actual accuracy, noise assumptions and runtime; no ranking of classical methods, and no fixed advantage in decibels, carries over across datasets and settings. A network can represent similar cross-channel predictions, but it chooses what to interpolate implicitly, which is one reason it is hard to say what it learned: a benchmark gain does not show that it learned this algorithm, and attributing the gain requires controlled experiments rather than an analogy between architectures (Chapter 15.6, Learned demosaicking).

Demosaicking as demultiplexing: frequency-domain methods🔗⧉

One family of methods treats the mosaic the way a radio engineer treats a signal. As Chapter 15.3, Mosaics and aliasing shows, a Bayer mosaic is exactly a luma component $L$ plus two chroma components modulated onto carriers at the highest frequencies the grid supports: $C_1$ on the horizontal and vertical carriers, $C_2$ on the diagonal one,

Demosaicking is then demultiplexing: isolate each modulated chroma band with a bandpass filter, shift it back to baseband by multiplying by its carrier, and take the luma as what remains. Alleysson, Süsstrunk and Hérault Alleysson, Süsstrunk & Hérault 2005 (frequency-domain demosaicing) arrived at this from the side of human vision, where luma and chroma are carried separately; Dubois Dubois 2005 (frequency-domain demosaicking) made it adaptive. Nowhere else in this chapter is the algorithm itself the frequency argument, rather than a heuristic that happens to be analyzable in frequency terms.

1. Diagonal chroma. Filter the mosaic with a bandpass filter centered on the diagonal carrier and multiply by $(-1)^{i+j}$ to bring $C_2$ back to baseband.
2. Axial chroma, twice. $C_1$ is carried twice, on the horizontal and on the vertical carrier. Extract it from each with its own bandpass filter and demodulate.
3. Choose the cleaner copy. Luma detail leaks into the two copies differently: a vertical texture puts its energy near the horizontal carrier and spoils that copy, while the vertical copy stays clean. Dubois weights the two estimates at each pixel by the local energy of the crosstalk in each, which is the edge-direction idea of this chapter moved into the frequency domain.
4. Luma. Remodulate the chroma estimates and subtract them from the mosaic; what remains is $L$.
5. Back to RGB. $G=L-C_2$, $R=L+C_2+2C_1$, $B=L+C_2-2C_1$.

# frequency-domain demosaicking (Dubois style), for an RGGB mosaic y
C2  = lowpass(bandpass(y, carrier=(pi, pi)) * alternate(i + j))
C1h = lowpass(bandpass(y, carrier=(pi, 0))  * alternate(i))
C1v = lowpass(bandpass(y, carrier=(0, pi))  * alternate(j))
eh, ev = local_energy(crosstalk_h), local_energy(crosstalk_v)   # luma leaking into each copy
C1  = (ev * C1h + eh * C1v) / (eh + ev)                         # trust the cleaner copy
L   = y - C1 * (alternate(i) + alternate(j)) - C2 * alternate(i + j)
G, R, B = L - C2, L + C2 + 2*C1, L + C2 - 2*C1

Figure 15.2.2⛶ fullscreen runs the same demultiplexing on the small inputs.

The filters are where the quality lives. Leung, Jeon and Dubois Leung, Jeon & Dubois 2011 (least-squares luma-chroma demultiplexing) design them by least squares on training images rather than by hand, the least-squares luma-chroma demultiplexing that is the version to cite for performance: a small learned element inside an entirely classical structure. The limit is the one Chapter 15.3, Mosaics and aliasing names: where luma and chroma overlap in frequency, as on fine texture near the carriers, no linear filter can separate them, and the leftover luma shows up as false color.

On the upper floors of a tower facade, where the window grid approaches the carriers, choosing the cleaner copy of $C_1$ per pixel removes much of the false color that averaging the two copies leaves, and the method scores above AHD (Figure 15.2.9⛶ fullscreen). The same implementation falls below bilinear interpolation on the flower of Figure 15.2.8⛶ fullscreen, whose saturated edges carry chroma at frequencies the low-pass filters remove.

Where the classical methods stop🔗⧉

Four situations defeat this family, and they are worth listing because the learned methods of Chapter 15.6, Learned demosaicking are judged on exactly them.

Diagonal edges can challenge a horizontal/vertical decision. More directions or a broader model can help; failure is not inevitable on every diagonal.

Repetitive texture near the sampling limit can alias into apparently consistent false structure. Wider context helps some cases, but a single mosaic may leave genuine ambiguity.

Clipped samples no longer report the incident signal linearly. Mark them and avoid treating their color differences as ordinary measurements. The amount of recoverable highlight color depends on which channels remain useful.

Noise perturbs gradients and can switch interpolation decisions. The error depends on signal-to-noise ratio, not ISO alone. Chapter 15.8, Joint demosaicking and denoising incorporates this uncertainty in reconstruction.

All channels at once: luma and chroma🔗⧉

The mosaic is one image, not three, and its spectrum is the sum of the three sampled channels. Because the masks add up to one at every pixel, their impulses away from zero cancel for a gray scene: whatever sits at a carrier frequency is a difference between channels. For Bayer, with red at even-even positions and blue at odd-odd positions, the algebra is exact and short:

At even-even positions the expression gives $R$, at mixed parity $G$, and at odd-odd positions $B$, which is a direct implementation check; a different origin for the array changes the signs of the carriers. The luma $L$ is a mixture of camera channels, not CIE luminance. Multiplication by $(-1)^i$ shifts a spectrum by $1/2$ cycle per pixel, so the mosaic is the luma at baseband plus $C_1$ modulated onto the two axis carriers $(1/2, 0)$ and $(0, 1/2)$, plus $C_2$ modulated onto the diagonal carrier $(1/2, 1/2)$ (Alleysson, Süsstrunk & Hérault 2005 (frequency-domain demosaicing); Dubois 2005 (frequency-domain demosaicking)). Given separated components, $G=L-C_2$, $R=L+C_2+2C_1$ and $B=L+C_2-2C_1$.

The luma is sampled at every pixel and keeps the full band of the pixel grid. The chroma is what pays for the mosaic: it is recorded only as copies parked at the highest frequencies the grid supports. This works because natural images have much less chroma detail than luma detail, so the chroma copies are compact and mostly stay clear of the luma band. Chapter 15.2, Advanced traditional demosaicking used this model as an algorithm, filtering the chroma copies out and shifting them back; here it is a way of seeing what goes wrong.

False color and moiré as spectral overlap🔗⧉

Aliasing on a mosaic is the luma band reaching a chroma carrier. Fine gray detail near $1/2$ cycle per pixel, a fence, a woven fabric, the fine print on a page, puts luma energy exactly where a demosaicker looks for chroma. The demosaicker cannot tell the two apart and reports a color the scene did not have: false color. A pattern whose frequency is close to a carrier, but not on it, beats against the carrier and produces a slow colored ripple: color moiré. Where the luma band and a chroma copy overlap, the mosaic does not determine which is which, and no reconstruction can separate them without an assumption about the scene. Image context and priors, including learned ones (Chapter 15.6, Learned demosaicking), make better guesses, and extra captures (Chapter 15.12, Hardware alternatives to the mosaic) remove the ambiguity; none of this recovers arbitrary overlapping signals from one mosaic.

Foreshortening makes this common. A fine pattern on a surface that recedes from the camera, a floor, a roof, a striped shirt turned away, grows finer on the sensor with distance, so it sweeps through every frequency on its way to the horizon and passes the carriers at some distance or angle, where the colors appear (Figure 15.3.3⛶ fullscreen).

The same picture explains the two greens. Green carries half the luma, so doubling it keeps the luma band wide, while red and blue enter mostly through the chroma, whose narrow band the sparser lattice can hold. This is a statement about natural images and about the eye, which also resolves luma more finely than chroma, and not about green light as such. A sharp colored pattern, red text on a blue sign, violates the assumption and aliases in the chroma itself.

The model is global and images are not. A photograph is smooth in some regions and finely textured in others, so a filter that separates the bands correctly in one place is wrong in another; the adaptive methods of Chapter 15.2, Advanced traditional demosaicking estimate the local structure and change the filter accordingly. The carrier positions describe an array, not the error of a particular demosaicker on a particular image Hirakawa & Wolfe 2008 (spatio-spectral sampling).

Learned demosaicking🔗⧉

Fine fabric and thin diagonal structures can defeat local interpolation rules. A learned demosaicker uses a larger training collection to fit how sparse color samples map to full-color detail.

A synthetic training pair is easy to construct by selecting one component per pixel from a full-color image. Producing a representative clean target is harder: a processed photograph may already contain demosaicking, denoising, sharpening and compression artifacts, and its RGB channels need not be camera-linear responses.

Specify the color domain and how the target was prepared before treating remosaicking as a sensor simulation.

Joint demosaicking and denoising🔗⧉

A noisy raw mosaic contains missing color components and uncertain measured samples. Demosaicking and denoising can therefore benefit from a shared estimator. A denoiser that ignores the CFA mixes unlike colors; a demosaicker that ignores noise can make poor direction decisions and spread their errors.

These are failures of particular models and interfaces, not proofs that either sequential order must fail. CFA-aware denoising can preserve color phases, and an RGB denoiser can model correlations introduced by demosaicking. Joint estimation retains access to the original observations and avoids treating an intermediate estimate as exact. Its benefit must still be measured against suitable sequential baselines.

For a fixed linear demosaicker $A$ and raw noise covariance $\Sigma$, the output covariance is $A\Sigma A^\mathsf{T}$. Reused samples correlate output values. Simple averaging of independent equal-variance samples reduces variance, but negative correction weights or noisy adaptive choices need not. Thus post-demosaic noise is not universally quieter, and a raw-domain scalar variance estimate does not fully describe it.

Follow a possible error on a striped shirt through a sequential pipeline. A demosaicker produces a patch of false color where its local model cannot resolve the texture. A subsequent denoiser may preserve that patch because it is spatially coherent, even while removing independent grain around it. Sharpening can then increase its contrast. Each operation can be reasonable for its assumed input while the composite result reinforces an earlier mistake. This is a reason to retain the original mosaic and its uncertainty when designing the estimator. It is not inevitable: a sequential method whose later stages model reconstruction artifacts can behave differently.

The information-flow issue is the commitment at an intermediate image. Once a stage passes only one RGB estimate onward, a later stage may have no indication that two different textures fitted the original mosaic almost equally well. It cannot reconsider that ambiguity using the raw likelihood unless the raw data or an adequate uncertainty representation remains available. A joint objective keeps those constraints active while the estimate changes. This is the useful distinction between coordinating several reconstruction operations and merely running them once in sequence.

The two orderings, made visible🔗⧉

Compare the same raw input, output domain and computation budget. Tune the sequential and joint methods separately on validation data.

Denoising before reconstruction can remove details needed for interpolation if it oversmooths them. A CFA-aware denoiser need not mix neighboring color channels, and its strength can be chosen to retain those structures.

Demosaicking first changes the noise covariance and may introduce structured artifacts. A denoiser trained or designed for this input can address some of them. Independent white-noise assumptions should not be transferred unchanged from raw samples to demosaicked RGB.

FlexISP: one energy, one prior🔗⧉

The idea of collapsing the pipeline into a single optimization was crystallized by Heide et al. (2014) in a system named FlexISP, a flexible image-signal processor built as one energy function rather than a fixed chain Heide et al. 2014. Its architecture is exactly the inverse-problem template we have been using. On one side, a single image-formation model: a linear operator $A$ that composes all the physical steps the sensor performed (subsampling by the color-filter mosaic, blurring by the optics' point-spread function, geometric warps for lens distortion and lateral color, even the resampling of a burst of frames) into one map from the clean latent image $x$ to the raw measurement $y$. On the other side, regularization applied to the shared latent image, combining image and cross-channel structure. Recovery is the one objective $\tfrac12\lVert Ax-y\rVert^2 + \lambda\,\Phi(x)$, minimized as one large optimization.

A composite operator permits the data fit to account for several physical effects together. Operator splitting decomposes the numerical solve into manageable subproblems; it does not remove the need to define each forward operation, boundary and adjoint consistently. Chapter 13.11, Denoising as a universal prior distinguishes explicit proximal priors from arbitrary inserted denoisers.

A shared final-image prior avoids some incompatible intermediate objectives, but its accuracy remains a modeling choice. FlexISP demonstrates a flexible formulation, not a theorem that one prior always beats every stagewise pipeline.

FlexISP evaluated ADMM Boyd et al. 2011 (ADMM) and a first-order primal-dual solver, and used the latter in its reported examples. Its regularization combines BM3D Dabov et al. 2007, total variation and a cross-channel term rather than relying on a color penalty alone (FlexISP paper). Splitting the numerical work into updates does not recreate a one-pass ISP: the updates revisit one shared image and reconcile their constraints through iteration. This is why a system can contain several computational modules while still solving a joint reconstruction problem.

Where demosaicking sits in the pipeline🔗⧉

Demosaicking normally follows linearization, black-level correction, and defect handling. White balance and denoising may occur before it, after it, or jointly with it; a scalar CFA sample can already receive its channel's white-balance gain. A camera-to-output color matrix requires reconstructed channels. Nonlinear rendering and output encoding belong to the later appearance stages, as summarized in Chapter 8.12, Recap ISP, non-destructive editing.

Pixel shift: spending time, precisely🔗⧉

In an ideal static scene, a four-shot cycle with one-pixel offsets can sample each scene-grid position through red, both green phases and blue. This avoids interpolating missing color components at covered interior positions. Actual fusion still needs calibration, accurate offsets, valid boundaries and motion handling.

Two patterns are in use and they buy different things. A one-pixel, four-shot cycle buys color: full color at the sensor's native resolution, direct sampling of each color at covered positions, with demosaicking removed from the pipeline entirely. A half-pixel cycle, usually eight or sixteen shots combining whole- and half-pixel steps, buys resolution as well, interleaving the sampling grids finely enough to raise the effective pixel count, which is the deliberate version of the hand tremor that Chapter 14.2, Multi-frame super-resolution exploits. Sony calls it Pixel Shift Multi Shooting, Pentax calls it Pixel Shift Resolution, OM System calls it High Res Shot, and Hasselblad calls it Multi-Shot, which tells you the market: landscape, product, and cultural-heritage work.

Commanded actuator offsets do not guarantee exact scene registration. Shake, lens distortion, actuator error and subject motion can perturb them. Fusion may reject moving regions or use a single-frame reconstruction there. Controlled pixel shift and unconstrained handheld fusion therefore share sampling principles but need not use identical algorithms.

Foveon and stacked photodiodes: spending silicon depth🔗⧉

Silicon absorbs short wavelengths near its surface and long wavelengths deeper in, so three photodiodes stacked at different depths in the same pixel respond to three different, broad, overlapping spectral bands. That is the Foveon sensor (Lyon & Hubel 2002 (Foveon)): three measurements at every pixel location, no filter array, no demosaicking, with reduced CFA-specific color aliasing when all channels share the same grid. Ordinary spatial aliasing and channel-response differences remain; some stacked designs also use different sampling densities across layers. The full account of the physics is in Chapter 4.6, Sensing color: multiplexing strategies; the reason it belongs here is what it implies for reconstruction, which is that missing-color interpolation can be reduced while spectral separation and noise estimation remain necessary.

It did not displace Bayer, and the reasons are instructive because none of them is about the idea. The three depth responses overlap heavily and are not the primaries anyone would choose, so recovering colorimetric values needs a color-separation transform whose coefficients and conditioning can amplify noise: the color separation is set by silicon's absorption curve rather than by a filter someone designed. Carriers generated at one depth diffuse to another, which is crosstalk that further mixes the bands. The potential advantage is collecting useful light at several depths without a surface color mosaic. Not every incident photon is collected, however, and collection efficiency differs by layer. The resulting benefit depends on both collection and how well the overlapping responses can be separated.

The lesson generalizes past this one sensor. A measurement that captures more per pixel is only better if the mixing it introduces can be inverted without amplifying noise, which is the same conditioning argument that governs every multiplexed measurement in Part 34, Computational sensors.

Four ways to multiplex color🔗⧉

The mosaic is a clever answer to a hard constraint (a photosite measures one number, and color needs three) but it is not the only answer, and seeing the alternatives makes clear what the mosaic trades away. Broadly, you can multiplex the three measurements in time, in space, or in depth.

• Temporal multiplexing: take three exposures through three filters, one after another. Sequential filtered acquisition was used by Sergei Prokudin-Gorskii for color photographs of the Russian Empire around 1900, shooting three plates through red, green, and blue filters. (The same idea lives on in the color wheel spun in front of a monochrome sensor.) You get three real measurements at every pixel and need no interpolation, but only for a static scene; anything that moves between exposures shows color fringes.
• Spatial multiplexing: one measurement per location, varying the color across the grid: the Bayer mosaic itself, and, not coincidentally, the strategy your retina uses (a mosaic of long-, medium-, and short-wavelength cones). Single sensor, high resolution, mature technology, at the cost of interpolation (this whole chapter), color jaggies, the resolution lost to the OLPF, and the light absorbed by the filters.
• Three sensors: split the incoming light with a prism and route it to three separate sensors, one per color (the 3-chip / 3-CCD video cameras, named after the charge-coupled device (CCD) sensor). Three real values per pixel and almost no photons wasted (the prism redirects rather than absorbs), but you pay in cost, bulk, and alignment.
• Depth multiplexing: stack the color-sensitive layers so each pixel measures all three at once at different depths: color film (tripack emulsions) and the Foveon sensor, which exploits the fact that longer wavelengths penetrate silicon more deeply. Three numbers per pixel and good light efficiency, but it needs more color processing and tends to be noisier.

These all tie back to Chapter 4.1, Linear color reproduction and CIE XYZ of the book; the takeaway is that the Bayer mosaic wins on cost, resolution, and single-sensor simplicity, and demosaicking is the computational price we pay for that win.

Lippmann and Gabor: recording the wave🔗⧉

The intellectual root is again Chapter 31.2, Light field cameras, the same physicist behind integral photography. In 1891 Gabriel Lippmann invented interference color photography Lippmann 1891, which won him the 1908 Nobel Prize. He recorded color not with dyes but with standing waves: light reflecting off a mirror behind the emulsion interfered with the incoming light, laying down microscopic layers in the film at spacings set by the light's wavelength, so the developed plate reconstructed color by reflecting back exactly those wavelengths. It was the first photographic process to record information in the standing-wave structure of light rather than in intensity alone, the conceptual seed of capturing phase.

Holography proper was invented by Dennis Gabor in 1948 Gabor 1948, for which he received the 1971 Nobel Prize. (He was trying to improve electron microscopy, not to make 3-D pictures; the 3-D display application came later.) Gabor's insight: bring the light from the object together with a coherent reference beam at the recording plate, and let them interfere. Where the two waves arrive in phase they reinforce (bright); where they arrive out of phase they cancel (dark). The resulting pattern of fine fringes is an intensity pattern an ordinary plate can record, but because the fringe positions depend on the phase difference between object and reference at each point, that intensity pattern secretly encodes the object wave's phase as well as its amplitude. The phase difference appears in a recordable intensity. With object field $U_o$ and reference field $U_r$,

The cross term carries their relative phase. A single intensity sample does not uniquely determine a general complex field; the known reference and the holographic encoding geometry make separation and reconstruction possible.

On replay, illuminate the developed fringe pattern with the reference beam again, and the fringes act as a diffraction grating. They bend the reference light by exactly the angles that reconstruct the original object wave. The wavefront that leaves the hologram is (to good approximation) the very wavefront the object once sent. Within the reconstructed aperture, wavelength band, and viewing zone, it can reproduce the corresponding visual cues: look from the left and you see the object's left side; move your head and it shows full, continuous parallax; and your lens accommodates to the true depth of each virtual point, because the reconstructed wavefront has the correct curvature.