6.8 When linearity breaks down⧉
A caveat: the clean equation $\mathbf{y}=A\mathbf{x}$ is an idealization with real edges. Five departures matter.
First, clipping (see Chapter 2.8, Noise, signal-to-noise ratio and dynamic range). A photosite well holds only so many electrons and an encoded pixel only so many code values, so the moment a measurement saturates (a blown highlight, a crushed black) the response stops being linear: past the limit, doubling the light no longer doubles the number, it returns the same ceiling (or floor). Clipping is a hard nonlinearity bolted onto the linear model, which is exactly why saturated pixels poison a deconvolution or an exposure merge and have to be detected and handled specially (→ Chapter 16.4, Blind deblurring, HDR merging). Distinguish this from the gamma nonlinearity of the encoding: gamma is invertible, so we simply undo it and work in linear light, whereas clipping destroys information outright and no inverse brings it back.
Second, as discussed in Chapter 4.3, Encoding: linear, gamma or log, images are usually stored with a non-linear encoding called gamma ($V = L^{1/\gamma}$ for linear light $L$), which must be undone before solving linear problems. Figure 6.8.1 compares inversion in linear light and encoded values. Applying a linear blur model to encoded pixels introduces a model mismatch whose visible and numerical effects depend on image contrast, blur, and regularization.
For an ideal power-law encoding $V=L^{1/\gamma}$, an exposure gain $2^{EV}$ becomes $V'=2^{EV/\gamma}V$, and a channel gain $g_c$ becomes $V_c'=g_c^{1/\gamma}V_c$. These identities are exact only for the power-law model before clipping. sRGB has both an offset power segment and a linear toe; use its actual decode and encode functions for an exact adjustment.
Quantization is different: rounding discards information. With code step $\Delta$, nearest rounding has error at most $\Delta/2$ before clipping. Modeling that error as uniform and independent, with variance $\Delta^2/12$, requires sufficiently varying or dithered input; a smooth gradient can instead produce structured banding. Its size relative to shot and read noise depends on gain, bit depth, and processing. Encoding also makes a constant code step correspond to a signal-dependent step in linear light. See Chapter 2.8, Noise, signal-to-noise ratio and dynamic range.
Third, not everything we want is a linear function of the scene. The model $\mathbf{y}=A\mathbf{x}$ recovers the image $\mathbf{x}$ (radiance on a grid), but much of computational photography is after something else entirely: the depth of each pixel, the motion in the frame, the material of a surface, the identity of a face. These are in general not linear read-outs of the light. Depth from defocus or stereo, for instance, enters through a ratio of blurs or a disparity search (a nonlinear estimation, not a matrix inverse) even though the image formation that produced the measurements was perfectly linear. The linear-operator picture governs how light becomes pixels; it does not promise that the thing you actually care about is linearly encoded in them. This is a recurring note of Chapter 5.8, Photographic representation and its limits: a photograph is a flat projection, and the quantities that projection discards, depth above all, must be inferred, not inverted.
Fourth, even where the forward model is linear, two practical gaps open around $A$ itself. Sometimes you simply do not know it: blind deblurring, the bilinear case of Chapter 6.7, Image formation as a linear system, must solve for the operator and the scene together, and is rescued only by priors on both (→ Chapter 16.4, Blind deblurring). More insidiously, sometimes you think you know $A$ but you are wrong: the point-spread function is slightly off, the noise is not quite Gaussian, the optics have drifted. A confident inversion of a subtly mistaken model manufactures confident artifacts, structure that looks real and is not. Robustness to model error is a real and under-appreciated cost of every method that leans hard on knowing $A$.
Fifth, a linear forward model can have either a linear or a nonlinear estimator. Fixed quadratic regularization gives, when the matrix is invertible,
which is linear in the measurement $\mathbf y$. A nonzero prior mean adds an affine offset. Sparsity penalties, positivity constraints, adaptive weights, and learned denoisers generally produce nonlinear estimators. The forward model and the estimator are separate choices (Chapter 7.1, Imaging as Bayesian inference with probabilistic priors).
A differentiable nonlinear forward model $F$ can also be approximated locally by $F(\mathbf x+\delta)\approx F(\mathbf x)+J_F(\mathbf x)\delta$. The Jacobian describes small changes around the current estimate, not a globally linear camera. Clipping has zero derivative beyond saturation; visibility changes and other discontinuities may invalidate this local approximation.
Repeated rounding to a low bit depth and strong shadow lifting make quantization particularly visible. A tone curve can spread a handful of input codes across a broad displayed interval, turning small code steps into bands. Dithering changes the spatial pattern of the error, while retaining adequate precision between operations avoids additional rounding. Neither recovers distinctions already removed by the first quantization.