15.7 Neighboring problems in the raw stage⧉
A raw pipeline must account for offsets, clipping, defects and spatial sensitivity variation before interpreting values as scene measurements. Some corrections precede demosaicking; others are fused with reconstruction or applied later with suitable models.
These operations have different statistical effects. Subtracting a calibrated offset changes the mean; a gain scales variance; interpolation mixes samples and can correlate their errors. Track these changes rather than treating every correction as the same operation.
15.7.1 Defective pixels⧉
Every sensor ships with defects, and collects more with age and radiation. Dead pixels read nothing, stuck pixels read a constant, hot pixels leak dark current and grow brighter with exposure time and temperature, and a few are simply out of specification.
They are found two ways. The factory measures them and stores a defect map on the device, which is fast and misses the ones that appear later. Or they are detected at capture, by comparing each photosite against its same-color neighbors and flagging outliers. Repair is interpolation from those same-color neighbors, which is demosaicking on a one-pixel hole.
The two failure modes are opposite and both common. A missed defect survives demosaicking as a colored speck, because the interpolation carries one wrong sample into the two channels it did not measure, so the appearance and extent of a defect depend on its CFA channel and the reconstruction. A hot green sample does not universally become magenta. An over-aggressive detector eats real content: a star in an astrophotograph is a bright isolated point and looks exactly like a hot pixel, so astronomy pipelines make this decision with dark frames rather than with an outlier test (Chapter 13.2, Denoising by averaging covers the calibration frames). Specular glints and single-pixel highlights lose the same argument.
15.7.2 Phase-detect autofocus pixels⧉
A modern sensor has autofocus built into the imaging array: a fraction of its photosites are partially masked, or share a microlens in pairs, so that each one sees light from one side of the lens aperture. Comparing the two sides gives the phase difference the autofocus needs, and the pixels are scattered across the frame in a regular pattern, sometimes a percent of the array, sometimes far more when every pixel is split.
Dedicated masked phase-detection sites can need calibrated gains or interpolation. Dual-pixel sensors are different: summing their two photodiodes supplies an imaging sample, so every split pixel is not a defective location to be patched. The correction depends on the sensor design, lens and readout mode.
A mismatched correction can leave a regular pattern that subsequent interpolation spreads. Later processing may reduce it, but preserving sensor-domain information makes diagnosis and correction easier.
For dedicated masked sites, the imaging cost is part of the autofocus design. Their directional sensitivity supplies focus information, but correcting their imaging response requires knowing how the lens illuminates them. A calibration that works for one lens or aperture may leave a pattern under another. The sensor has gained a measurement capability, and the reconstruction must account for the changed samples.
15.7.3 Black level, clipping, and where pink highlights come from⧉
A raw file's zero is not zero. Sensors are biased so that noise can go both ways without truncating, so a black-level offset must be subtracted before anything treats the values as light. Subtract the wrong one and the shadows tint, and the tint survives every later stage.
The other end is worse. Each channel clips at its own point, and after white balance those points are at different signal levels, and which channel saturates first depends on the scene spectrum, exposure, gains and sensor response. The pixel's ratios are then wrong, and demosaicking, which assumes it is interpolating a linear signal, propagates them into the neighborhood: the center of a clipped highlight goes pink or cyan and the artifact is larger than the clipped region.
Record the clipping thresholds and a saturation mask before transformations. Highlight recovery estimates missing information from unclipped channels or neighboring structure; it cannot guarantee original color where the useful measurements have saturated. Its placement depends on what the reconstruction and rendering stages expect.
15.7.4 Lens shading and crosstalk⧉
Lens shading is the fall-off in brightness toward the corners, from the lens's own vignetting and from the geometry of light striking a photosite off-axis. It is corrected by a smooth per-channel gain map, per lens and per aperture, and the color half of it matters as much as the brightness half, since the three channels fall off differently and an uncorrected difference is a color cast toward the corners.
A known gain $c_p$ multiplies noise standard deviation by $|c_p|$ and variance by $c_p^2$ relative to that pixel before correction. It does not by itself specify the corner-to-center noise ratio. Update the noise model using the pre-correction signal and noise, as Chapter 13.1, Noise models, transforms, and benchmarks derives.
Crosstalk can mix light or charge between sites. The effect on noise depends on the mechanism: deterministic mixing of measured values creates covariance, while independently redirecting photons need not produce the same covariance model. A color matrix can compensate some spectral mixing, but cannot generally invert spatial blur. Characterize spatial and spectral response separately.
15.7.5 The ordering claim, once more⧉
When an artifact appears, inspect the raw samples and metadata before assigning a cause. Some appearances suggest where to look first. An isolated colored speck a few pixels across that sits in the same place in every frame points to a defective photosite. A faint regular grid in flat regions points to autofocus pixels or another fixed pattern. Pink or magenta in the center of a bright highlight points to channels clipping at different levels after white balance. A color cast that grows toward the corners, different for each lens or aperture, points to shading. Defects, autofocus corrections, clipping and shading can all contribute at once, so these are leads, not a unique diagnosis. Preserve validity masks and update the noise statistics after gains or interpolation. Joint estimation is useful when stages otherwise discard information needed later; it still needs a correct forward model.