15.13 Demosaicking TLDR🔗⧉

• Preserve the raw mosaic, CFA layout and origin, black/white levels, gain and saturation mask. Cropping or rotating a mosaic changes its phase. A network expecting RGGB will not interpret BGGR correctly by itself.
• Use a demosaicker matched to the sensor and noise conditions. Compare a strong classical method with a learned or joint method on fine fabric, colored edges and small text; clean-looking detail can still be wrong (Chapter 15.1, Basic traditional demosaicking, Chapter 15.2, Advanced traditional demosaicking, Chapter 15.6, Learned demosaicking).
• Model noise in the domain where reconstruction operates. Gains scale variance by their square; interpolation can introduce correlations. Joint demosaicking and denoising can help, but compare it with properly designed sequential baselines (Chapter 15.8, Joint demosaicking and denoising).
• Check calibration before tuning reconstruction: defects, autofocus-site correction, clipping, shading and chromatic displacement can create colored artifacts. Highlight recovery estimates missing signal; it cannot guarantee original color after saturation.
• Keep a documented color representation for later rendering. Learned reconstruction can coexist with exposure, white-balance and tone controls. End-to-end training does not require giving up editable stages (Chapter 15.9, Joint methods and the end-to-end pipeline).
• Treat synthetic remosaicking and unprocessing Brooks et al. 2019 (Unprocessing) as training models. They do not recreate the original sensor capture exactly. Validate on representative cameras and report input domain, reference preparation, color error and computation.

A Bayer photosite measures one filtered channel, leaving the other two to be estimated. Two classical ideas make a useful starting point: interpolate along an edge, and interpolate color differences rather than independently smoothing every channel. Estimate the more densely sampled green channel, then use it to guide red and blue. Color differences are often smoother than the channels, though saturated color edges can break that assumption. Residual interpolation Kiku et al. 2013 (residual interpolation) takes the same idea further: fit a local relationship between channels and interpolate what the fit leaves unexplained (Residual interpolation and ARI).

The frequency view explains why fine monochrome texture can turn into false color. A periodic color-filter array shifts combinations of color signals into different frequency bands. Reconstruction must separate them; when scene detail overlaps a chrominance band, the measurements admit competing color and texture explanations. That is a sampling ambiguity, and a cleaner-looking reconstruction can still choose incorrectly (Chapter 15.3, Mosaics and aliasing).

Array design changes the problem. X-Trans changes the pattern and distribution of aliases; it does not eliminate aliasing. Quad-Bayer groups like-colored sites, supporting binning and reconstruction modes with different resolution and noise tradeoffs. RGBW adds broad-band measurements while reducing the sites allocated to filtered color. Match the reconstruction to the actual array and operating mode (Chapter 15.5, Alternative color filter arrays).

Denoising and demosaicking affect one another. Denoising a raw mosaic must respect the different channels; indiscriminate neighborhood averaging mixes different measurements. Demosaicking first can turn independent sensor errors into correlated errors and can turn a noisy direction decision into a visible artifact. Joint methods use the mosaic and its noise model together. They can help avoid this propagation, but still impose a prior and can lose detail or fail on unfamiliar data. A sequential baseline designed for the correct domains remains a useful comparison (Chapter 15.8, Joint demosaicking and denoising).

For a learned method, realistic supervision matters as much as the network. Smooth regions dominate random crops, while the difficult cases are fine fabric, roof tiles, fences, and colored edges. Include those cases during training and evaluation. Synthetic remosaicking provides convenient pairs, but a processed reference may already contain demosaicking artifacts. Unprocessing approximately reverses encoding and rendering to construct raw-like training examples; it cannot undo clipping or retrieve information discarded by the original pipeline (Chapter 15.6, Learned demosaicking, Chapter 15.11, Synthetic data, unprocessing, and raw reconstruction).

Inspect the artifacts separately from an average score. CPSNR can summarize reference error while hiding a small region of conspicuous zippering. Color and spatially filtered color-difference measures answer complementary questions; none replaces looking at the difficult structures. Use the same color representation, crop, alignment, and reference preparation when comparing methods.

The surrounding pipeline also matters. Learned raw reconstruction can feed a documented color representation with adjustable exposure, white balance, and tone controls. Joint training does not require an uneditable raw-to-JPEG network. Preserve the intermediate quantities a photographer or later rendering stage needs (Chapter 15.9, Joint methods and the end-to-end pipeline).

Alternative capture hardware changes which colors are missing. Pixel shift can measure more color samples at corresponding scene positions when the scene stays still. Layered sensors separate spectral responses through depth but still need color unmixing. Multi-sensor optical systems trade alignment and hardware complexity for simultaneous measurements. These alternatives spend time, space, or optical complexity to reduce the spatial sampling compromise of a mosaic (Chapter 15.12, Hardware alternatives to the mosaic).