15.1 Basic traditional demosaicking🔗⧉

This chapter covers the hand-designed reconstructions, the ones that encode their assumptions about images in the algorithm itself. They start from the layout of the Bayer tile Bayer 1976: the two green-filtered photosites of each 2×2 block sit on one diagonal, red and blue on the other (Figure 15.0.1, at the start of Part 15, Demosaicking and joint methods). Each filter has a broad spectral response, so a photosite records a camera-color component, not a wavelength.

Green gets the extra samples because camera green correlates strongly with brightness, where the eye wants the most detail. It is not colorimetric luminance, though; Chapter 15.3, Mosaics and aliasing makes the split between a combined brightness signal and color explicit.

The crux of the whole problem is that the three colors are measured in different places. Red sits on one sparse grid, blue on another shifted by (1,1) pixel diagonally, green on a denser checkerboard, and no two of them coincide. When the scene has a sharp edge, the three channels cross it at slightly different pixels, and such misregistration is the source of most demosaicking artifacts: zippering, false color and color moiré all come from it, while noise amplification and aliasing already present in the samples have other causes.

We will illustrate demosaicking with a scene composed of one dark corner on a light background (Figure 15.1.1⛶ fullscreen). It starts from the mosaic itself.

Test a synthetic constant field first, including unequal constants such as $(0.2,0.4,0.7)$: a correct reconstruction returns each channel's own constant, including at the borders. A real neutral target need not give equal raw channel values before white balance. Then use an achromatic rectangle and a colored edge, and test all four Bayer phases. Those cases expose indexing errors, zippering and failures of the cross-channel assumption.

Keep in mind where demosaicking sits in the pipeline. Raw values are linear in light, and demosaicking works on them in that form; the color-difference methods below depend on it. Its output is therefore still linear camera RGB, not a picture ready for a screen. Before display, it still needs the color matrix to a standard RGB space, usually a tone curve that gives the camera its look (the beauty-curve chapter), and the gamma encoding the display system expects (Chapter 4.3, Encoding: linear, gamma or log). Shown directly, a linear demosaicked image looks dark and flat.

15.1.1 Downsampling and bilinear interpolation, and how they fail🔗⧉

Two simple reconstructions come first, and their failure illustrates the demosaicking challenge.

The simplest is not to interpolate at all. Make each 2 × 2 tile one pixel, taking its red, the mean of its two greens and its blue. Every output value is a measurement, the two greens are averaged, which lowers their noise, and the operation costs almost nothing, which is why raw decoders such as dcraw offer it as a half-size mode and fast previews often use something like it. The first price is resolution: the image has half as many pixels in each direction, so detail that the sensor recorded is thrown away. And downsampling unfortunately does not remove the underlying problem: The red, green and blue of a tile were still measured at different positions, so a tile that an edge cuts in two combines red from one side with blue from the other and gets a color the scene does not have (Figure 15.1.1⛶ fullscreen, stage downsample).

The same happens on real pictures and on test charts, where the comparison with the scene is side by side (Figure 15.1.2⛶ fullscreen, stage downsample).

Bilinear interpolation keeps the full resolution instead. It treats the three color channels as three independent, sparsely sampled images and interpolates each one. For a red pixel missing green, average the four green neighbors; for a green pixel missing red, average the two horizontal or two vertical red neighbors, and so on.

It is a useful inexpensive baseline. It reproduces some smooth signals well, but its errors on fine structure illustrate the consequences of sparse color sampling.

Zippering is an alternating light-dark pattern along a sharp horizontal or vertical edge. It comes from averaging across the edge: at one pixel the four neighbors are two on each side, at the next they are three and one, so the interpolated value alternates. The name comes from what it looks like. The zipper is a form of aliasing (Figure 15.1.3⛶ fullscreen). Green's samples lie on a checkerboard, and a checkerboard is itself a regular square grid, of spacing $\sqrt2$ pixels and turned by 45 degrees, a pattern called a quincunx. To that turned grid, a horizontal or vertical edge of the scene is a diagonal line, and a regular grid samples a diagonal line as a staircase, the same jaggies a diagonal shows on a screen. Bilinear interpolation reconstructs the staircase rather than the edge, and the steps of the staircase are the zipper's teeth.

False color is a colored fringe on fine detail that is gray in the scene. It happens because the three channels are sampled at different positions, so a thin structure lands on a red pixel and misses the blue ones, and the interpolation reports a color the scene did not have.

Maze-like patterns often involve unstable direction choices in adaptive methods. They should not be attributed to a bilinear filter changing its interpolation direction.

These artifacts can involve aliasing, cross-channel disagreement and interpolation rules. Looking at their appearance helps diagnosis, but does not uniquely identify the algorithm or cause.

Concretely, for green, whose measured samples lie on a checkerboard, the four nearest measured neighbors sit above, below, left and right:

Red and blue are sparser, so the geometry varies: at some empty sites the nearest red samples are the four diagonal ones, at others the two horizontal or two vertical. The principle is the same, and it is a small tent kernel applied per channel.

The zipper is worth doing with numbers once. Take a vertical black-and-white edge, value 0 against value 1, and look only at green. At a missing green site just on the white side, three green neighbors can be white and one black, giving 0.75 instead of 1, while the measured green site beside it stays at 1. The reconstruction therefore alternates between measured and interpolated positions, which is exactly the run of teeth. On the black side the same averaging lifts missing values above zero.

The corner showed the same numbers along a whole edge (Figure 15.1.1⛶ fullscreen, stages bilinear-green and bilinear-rb).

15.1.2 Interpolate along the edge, not across it🔗⧉

The first classical idea: decide, per pixel, which direction is safe to interpolate in.

For a missing green value, compute a horizontal and a vertical gradient from the available samples, and interpolate along whichever direction varies less. The test goes back to Robert Hibbard's Kodak patent Hibbard 1995 (Kodak patent): compare the differences between the green neighbors horizontally and vertically, and interpolate horizontally, vertically or from all four neighbors depending on which gradient falls below a threshold. Hamilton and Adams's method (Hamilton & Adams 1997) does this with a test that uses both the green neighbors and the same-color neighbors two pixels away, which is what lets a red pixel estimate the second derivative of the red channel and use it to correct the green interpolation.

The second term is the correction that makes this work, and it is the second classical idea arriving early: it uses the red channel's curvature at the center to sharpen the green estimate, on the assumption that the channels share their structure.

Variants differ in how the direction is chosen. Variable number of gradients (Chang, Cheung & Pan 1999 (VNG)) computes gradients in eight directions and averages the ones below a threshold, which handles corners and diagonals better than a two-way choice. Adaptive homogeneity-directed interpolation (Hirakawa & Parks 2005 (AHD)) reconstructs the image both ways, converts both to a perceptual color space, and keeps the version that is locally more homogeneous in color, which is a defensible criterion because the eye tolerates luminance detail and not chrominance noise.

The plain two-direction form of the same idea is worth seeing before the correction term arrives:

Interpolate along whichever axis its two neighbors look most alike, on the bet that the other axis crosses an edge. The bet rests on the world being piecewise smooth: mostly flat regions separated by a relatively small number of strong, roughly one-dimensional edges. Get those right and the cases the eye notices are handled. A tie can take the four-neighbor average rather than preferring an axis, and continuous directional weights avoid abrupt switches. Fine periodic texture can make both directional differences misleading, and noise can flip the choice.

On the corner, the rule picks the right pair at every cell (Figure 15.1.1⛶ fullscreen, stage edge-green).

# edge-directed green interpolation, the Hamilton-Adams test
dH = abs(G[x-1,y] - G[x+1,y]) + abs(2*R[x,y] - R[x-2,y] - R[x+2,y])
dV = abs(G[x,y-1] - G[x,y+1]) + abs(2*R[x,y] - R[x,y-2] - R[x,y+2])
if   dH < dV: G_est = (G[x-1,y]+G[x+1,y])/2 + (2*R[x,y]-R[x-2,y]-R[x+2,y])/4
elif dV < dH: G_est = (G[x,y-1]+G[x,y+1])/2 + (2*R[x,y]-R[x,y-2]-R[x,y+2])/4
else:         G_est = average_of_both

This pseudocode is for a red center; exchange red and blue at a blue center. Copy measured green samples in noiseless interpolation and use a CFA-aware boundary extension that preserves color parity. Malvar–He–Cutler provides a useful fixed linear alternative with cross-channel Laplacian corrections (Malvar et al. 2004); it differs from selecting a direction per pixel.

Edge-directed green is the simplest case of a more general idea, steered reconstruction (Chapter 6.5, Resampling and upsampling). Reconstructing a sampled image means choosing, at each output pixel, weights for the nearby samples: a reconstruction kernel. A fixed kernel, like bilinear's, uses the same weights everywhere, so it blurs across edges and aliases along them. A steered kernel is shaped by the local image, long along an edge and narrow across it, so it averages where the signal is smooth and keeps the change sharp (Figure 6.5.19⛶ fullscreen). The two-direction green rule is its coarsest version: two candidate kernels, one horizontal and one vertical, and a gradient test to choose between them. The variants above refine the choice. VNG considers eight directions, AHD compares whole reconstructions, and the Hamilton–Adams correction adds information from the other channels to the kernel it chose.

Working out the local reconstruction weights from the local pattern is the key idea of traditional demosaicking, and it reaches beyond it. Kernel regression makes the steering continuous, estimating an orientation and an elongation at every pixel from the local gradients (Chapter 10.7, Locally adaptive regression kernel (LARK)). RAISR learns a bank of filters and picks one per pixel by hashing the local gradient pattern (Chapter 14.4, Single-image super-resolution: classical methods); applied to a mosaic, the same scheme would amount to a learned gradient test. None of these methods writes down an image prior, yet each encodes one in the rule that chooses the weights: that the image is smooth along some direction at every pixel. Chapter 15.6, Learned demosaicking replaces the hand-written rule with one fitted to data.

15.1.3 Interpolate differences, not channels🔗⧉

The second classical idea does more work than the first and is less often stated plainly.

Color differences can vary more slowly than individual channels, especially near neutral structure. They are not invariant to shading: if $R=rI$ and $G=gI$ for a common shading signal $I$, then $G-R=(g-r)I$, which varies unless $g=r$. The ratio $R/G=r/g$ is constant under this simplified model, but becomes unstable near zero green and changes at material or illumination boundaries.

So interpolate the difference. Estimate green everywhere first, since it is sampled twice as densely, then form $G - R$ at the red pixels, interpolate that smooth quantity, and recover red as $\hat R = \hat G - \widehat{(G-R)}$. The hard, high-frequency part of the problem is solved once, in the channel with the best sampling, and the other two channels inherit it.

On the corner, the differences are formed at the red and blue sites (Figure 15.1.1⛶ fullscreen, stage differences).

Green-first reconstruction is common, but other joint and frequency-domain methods do not follow that ordering. Difference interpolation fails when the assumed shared structure is weak: strongly colored edges, channel clipping and chromatic displacement are important cases.

As a recipe, in four steps:

1. Reconstruct green everywhere, using the edge-directed method above, since green has the densest sampling.
2. Form the difference $R - G$ at every pixel where red was measured, and $B - G$ at every measured blue pixel, which is possible now that green exists everywhere.
3. Interpolate the difference with plain linear interpolation and no edge logic, because the difference is usually the smoother quantity. Sharp color boundaries remain the failure case.
4. Add green back: $R = (R-G) + G$, and likewise for blue.

An achromatic edge has zero true color differences, so this keeps it gray, but only if the reconstructed green is also accurate at the measured red and blue sites. Green errors leave residual false color even on that simple target.

Even green-based, edge-directed demosaicking is not perfect. Fine, high-frequency color texture, a tiled roof, a striped shirt, a picket fence, breaks the smooth-color assumption and produces the residual false color often called maze or labyrinth artifacts, because at those scales the channels really are doing different things. That is where the modern methods earn their keep.

Where the maze comes from shows up best on a small patch of fine texture (Figure 15.1.9⛶ fullscreen). At a pixel where the horizontal and vertical tests come out nearly equal, a little noise decides which one wins, so neighboring pixels choose differently, and the pattern of choices is the labyrinth.

A red flower against green leaves is a useful counterexample to the shared-edge assumption: the boundary can have very different contrast in red and green. A clipped specular highlight breaks the linear relationship for another reason, because one channel stops increasing while another still responds. Lateral chromatic aberration breaks it geometrically by placing the same boundary at different coordinates. These cases need different remedies even when their demosaicked outputs all show colored fringes.

15.1.4 Demosaicking and other aspects of imaging🔗⧉

Demosaicking is one part of an imaging system. The optics and the sensor decide what the mosaic records, and the stages of the camera pipeline around it (Part 8, Basic image processing and ISP) decide what it is handed and what happens to its output. Each method in this chapter and in Chapter 15.2, Advanced traditional demosaicking assumes something about its input, and these other parts decide whether the assumption holds.

The anti-aliasing filter. Many cameras place an optical low-pass filter in front of the sensor. It blurs the image slightly before sampling, so that less fine detail aliases into color moiré, the failure no demosaicker can fully undo because the mosaic cannot tell it from real color. Many recent cameras have weakened or dropped it and leave moiré to the demosaicker. Chapter 15.3, Mosaics and aliasing explains what the filter can and cannot remove, and why it has been disappearing.

Grouped layouts: Quad and Nona. Phone sensors that group same-color photosites in 2 × 2 (Quad-Bayer) or 3 × 3 (Nona-Bayer) blocks change what a demosaicker receives. In low light the sensor bins each block and hands over an ordinary Bayer mosaic at a quarter or a ninth of the pixel count, and the methods of this chapter apply unchanged. At full resolution the same color is repeated over neighboring sites, so red, for example, is missing from larger regions than in a Bayer mosaic, and the nearest red sample can be three photosites away. A Bayer demosaicker applied directly to that layout uses the wrong colors at most sites; the usual route is to remosaic the grouped layout into an estimated Bayer mosaic first, which is itself an interpolation, then demosaick, or to demosaic the grouped layout directly (Chapter 15.10, Quad-Bayer and Nona-Bayer processing).

Color routers. A nano-prism or metasurface color router keeps the Bayer layout but steers each band of light toward the photosite that measures it instead of absorbing it (Chapter 4.6, Sensing color: multiplexing strategies). Each photosite then collects light from a wider area than its own, which acts as a mild anti-aliasing filter (Chapter 15.3, Mosaics and aliasing), and part of each band lands on neighbors of the wrong color, which broadens and mixes the channels' spectral responses. The demosaicking algorithms still apply, but the color correction that follows has to undo more mixing.

Black level. The sensor adds an offset to every sample, and it has to be subtracted first. Color differences and ratios are meaningful only on values proportional to light: with the offset left in, a gray edge no longer has a constant $R-G$ across it, and the ratio model of the "difference or ratio?" sidebar is wrong everywhere.

Defects and lens shading. A hot or dead photosite is one wrong sample, and interpolation copies it into its neighbors' missing colors, so a single bad value becomes a colored speck several pixels across; repair it on the mosaic. Lens shading darkens the corners by a different amount in each channel, which demosaicking would read as color. Both are covered in Chapter 15.7, Neighboring problems in the raw stage.

White balance. The color-difference methods work best when a neutral surface has equal values in all three channels, because then the difference across a gray edge is zero on both sides and interpolating it is exact. Raw values under most lights are far from equal for a neutral, so many pipelines apply the white-balance gains to the mosaic before demosaicking (Chapter 8.10, Auto-exposure and auto white balance). The gains then scale clipped samples too: a highlight that clipped in green is no longer at a common ceiling once red and blue are multiplied, which is where pink highlights come from, so saturated samples need to be marked before the gains are applied.

Chromatic aberration. Lateral chromatic aberration shifts the color channels relative to one another toward the edges of the frame. The cross-channel assumption then fails exactly at edges, where it is needed, and a demosaicker reads the misregistration as a colored edge. Correcting it before or during demosaicking avoids fringes that are much harder to remove afterward (Chapter 22.2, Aberrations correction).

Noise. Interpolation mixes neighboring samples, so independent sensor noise becomes spatially correlated color noise, which is harder for a denoiser to recognize. Noise also flips the edge-direction decisions of this chapter. Denoising the mosaic first, or demosaicking and denoising together, avoids both (Chapter 15.8, Joint demosaicking and denoising).

After demosaicking. The color matrix, tone curve and sharpening all act on the reconstructed image, and sharpening in particular amplifies zippering and maze patterns along with real detail (Chapter 8.8, Sharpening). A demosaicker that leaves small artifacts can look acceptable until the rest of the pipeline has run, so judge it on the finished rendering as well as on its direct output.

The practical rule is to know which of these steps have been applied to the data a demosaicker receives, in what units, and to tell the algorithm.

And all of it should happen on linear values. Black-level subtraction, defect repair, lens-shading and white-balance gains, and demosaicking itself operate on numbers proportional to light, and the image becomes nonlinear only afterward, when the tone curve and the gamma encoding prepare it for display. The reasons are the ones this chapter keeps meeting. Interpolation averages, and an average of gamma-encoded values is not the encoding of the average light, so a demosaicker run on encoded values darkens fine detail and shifts colors at edges (Chapter 6.5, Resampling and upsampling). Color differences and ratios describe the scene only when the values are proportional to light. And the noise model that joint methods rely on is simple only in linear units, before a curve stretches the shadows and compresses the highlights.