7.1 Imaging as Bayesian inference with probabilistic priors⧉
The linear-methods part modeled imaging as an inverse problem and studied its hardness using the tools of linear algebra:
Here $\mathbf{x}$ is the scene or image we actually want, $A$ is the imaging operator (a blur, a projection, a sub-sampling, a mask) and $\mathbf{n}$ is noise (see Chapter 2.8, Noise, signal-to-noise ratio and dynamic range). This inverse problem underlies most imaging and restoration tasks in the book. Deblurring, super-resolution, demosaicking, inpainting, denoising, tomographic reconstruction, and light-field reconstruction fit this model when the geometry and response are known. Depth from defocus instead requires estimating parameters that change $A$ (Figure 7.1.1).
We saw that the backward direction is hard because information is lost in two separate ways. The first is the noise: the $+\mathbf{n}$ term means that even a perfectly invertible $A$ hands back a corrupted answer, because we are inverting $\mathbf{y}$, not the clean $A\mathbf{x}$. The second is the operator itself: a real $A$ is almost never a clean, invertible square matrix. It can be wide (fewer measurements than unknowns) in which case it literally throws scene degrees of freedom away; or it can be square but ill-conditioned: it squashes some components of the scene almost to zero, so that undoing it means dividing by almost nothing, which detonates whatever noise was sitting there. Either way, the naive instinct ("just multiply by $A^{-1}$") is a catastrophe.
There are two useful formulations of recovery. Regularization adds a penalty to a data mismatch. Bayesian inference combines a measurement likelihood with a prior, giving a posterior distribution. Maximizing that posterior becomes a regularized optimization after taking negative logarithms. Linear observations with Gaussian noise and a Gaussian prior give quadratic regularization as a special case; other priors give other penalties.
Denoising isolates the role of statistical uncertainty. With the identity forward operator, there is no blur to invert, but we still do not know which part of the observation is signal and which is noise. A Gaussian likelihood favors candidates close to the observation; it does not assign equal plausibility to every image. A smoothness or learned prior can prefer a nearby candidate that removes a fluctuation judged unlikely in a clean image. Increasing the assumed noise level broadens the likelihood and lets that preference move the estimate farther from the data. The operator can be perfectly invertible while the clean-scene inference remains uncertain.
7.1.1 Priors and the manifold of natural images⧉
The data often leaves uncertainty even when the forward operator is simple. In denoising, $A=I$ and independent Gaussian noise gives a likelihood that decreases with squared distance from the observation. Candidates at equal distance have equal likelihood; candidates at different distances do not. The maximum-likelihood estimate is the noisy observation itself. To improve on it, we use structure shared by plausible clean images.
Fortunately, we are almost never after arbitrary images. We want pictures of the world around us, and such images are visually special, sharply distinguished from random arrays of numbers. Press sample random in Figure 7.1.2 and you draw a point uniformly from the space of all images: it will almost certainly look like television static. Structured photographs occupy a minute fraction of the bounded pixel-value space. The vast majority of arrays of RGB values look like noise, and look nothing like the photographs anyone actually wants to take.
If we regard an image as a high-dimensional vector in the abstract space of all images (the same space as the linear-algebra section above, one axis per pixel) then the natural images occupy only a tiny, structured region of it. We call that region the manifold of natural images (Figure 7.1.3): the thin, curved sliver where real photographs live and a good prior $p(\mathbf{x})$ is high, adrift in the overwhelming bulk of the space, which is noise and where $p(\mathbf{x})\approx 0$. Do not lean too hard on the word manifold: it is a useful mental picture more than a precise claim, and the locus of natural images very likely does not satisfy the strict properties of a mathematical manifold Kadkhodaie, Guth et al. 2024.
Natural images occupy only a tiny fraction of the space of all arrays of RGB values.
An image prior assigns different weights to candidate clean scenes. With an improper constant prior over unbounded pixel values and additive Gaussian noise, the posterior is centered on the noisy input and its mean is that input. A bounded uniform prior can behave differently near the bounds. A more informative prior can reduce estimation error on average under its model, but cannot guarantee the true image for every observation.
If we can computationally model the manifold of natural images, we can use it to exclude unlikely solutions to inverse problems.
A prior reduces ambiguity according to its assumptions. It can also favor the wrong answer when the scene differs from those assumptions.
When plausible solutions differ in edge location, their posterior mean can have a blurred or doubled edge. This estimate minimizes expected squared error, but need not resemble a typical draw from the posterior. A posterior sample commits to one possibility and can look sharper while being less accurate for the particular scene. A sharp edge at the wrong position is still wrong.
Denoisers are useful beyond denoising because their responses encode learned image statistics. Under particular assumptions, an ideal Gaussian denoiser relates to the gradient of a smoothed image distribution; Chapter 7.5, Image priors gives that relation. An arbitrary denoiser need not be a projection, the gradient of a scalar penalty, or compatible with every solver. Diffusion uses noise-conditioned predictions inside a specified sampling process, not an arbitrary denoiser applied repeatedly.
Restricting candidates to plausible images can still leave ambiguity. A strongly blurred grass field may have many plausible blade arrangements whose measurements are nearly indistinguishable at the available noise level (Figure 7.1.6). A prior can prefer grass-like textures without identifying the original blade positions.
We are not flying blind about the manifold's shape; decades of work have measured its regularities, and they are exactly what priors exploit. Natural images have $1/f$ amplitude spectra, a scale-invariance that says structure exists at every scale [@field-1987; @ruderman-bialek-1994]. Their gradient and wavelet statistics are heavy-tailed: mostly near-zero (smooth regions) with rare large values (edges), which is the precise meaning of sparsity and the basis of sparse-coding models of early vision Olshausen & Field 1996. They have strong local correlation. There is even a strikingly simple generative model that produces all of this at once, the dead-leaves model: opaque shapes of random size and brightness dropped one atop another, with a power-law distribution of sizes, so that occlusion alone reproduces the $1/f$ spectrum, the scale-invariance, and the heavy-tailed edge statistics (Matheron's random-set geometry; Lee, Mumford & Huang 2001 Lee, Mumford & Huang 2001 (dead leaves); Ruderman 1997 Ruderman 1997). It is the classic bridge from measuring image statistics to modeling them, and the hand-built ancestor of today's learned generative priors (diffusion). Each of these regularities is, quite literally, what a classical regularizer encodes (total variation bets on sparse gradients, wavelet priors on sparse coefficients) and the whole program of reading images off their statistics is a well-developed science [@simoncelli-olshausen-2001; @hyvarinen-etal-2009].
Hand-written priors describe selected regularities, such as sparse gradients. Learned models can capture richer dependencies, but their distribution reflects training data and is not a near-perfect description of all possible photographs. Generative models provide ways to sample from such learned distributions (Chapter 7.4, Generative AI and diffusion).
Assumptions enter through explicit penalties, architecture, training examples, and the training loss. These choices coexist: a convolutional network still imposes locality and weight sharing while learning from data. Sutton’s Bitter Lesson argues for methods that benefit from increasing computation; it does not guarantee that a larger model beats a calibrated method on every imaging task. The lesson is about what wins at scale, not about what is understandable or data-efficient. Hand-built priors and architectural constraints remain the right tool when data are scarce, when guarantees matter, or when you must know why an answer came out as it did: a total-variation penalty states its assumption in one line, can be checked against the physics, and needs no training set, while a learned prior can encode far richer structure but inherits the gaps and biases of the data it saw.
A bilateral filter, an edge-directed demosaicker, and an end-to-end network each bet on what natural images look like. One states the bet as a penalty, one hides it in the machinery, one absorbs it from data, and none of them can avoid making it, because the measurement alone does not determine the answer. The only choice is whether the bet is deliberate and inspectable. Faced with any method in the chapters ahead, locate its prior first: it tells you which images it will quietly fail on.
Levin and colleagues study how image statistics limit denoising and what larger image neighborhoods can contribute [@levin-nadler-2011; @levin-etal-2012-patch]. Their work addresses the quantitative question behind the manifold picture: how much uncertainty remains once the model has used the statistical evidence? A plausible-image constraint can reduce the set of candidates substantially while still leaving uncertainty about the actual texture.
7.1.2 Probability: the Bayesian view⧉
Bayes’ rule combines the measurement with prior knowledge to give a distribution over candidate scenes:
Thomas Bayes (c. 1701–1761) was an English Presbyterian minister whose single posthumous paper (read to the Royal Society in 1763 by his friend Richard Price) gave us the rule for updating a belief with evidence: posterior $\propto$ likelihood $\times$ prior. That one line is the backbone of the MAP estimate above, of the priors that regularize every ill-posed reconstruction in this part, and of the Bayesian reading of denoising, white balance, and blind deblurring elsewhere in the book. Even this portrait is of doubtful authenticity; it may not be Bayes at all. Portrait: 1936 reproduction of disputed origin, public domain (via Wikimedia Commons).
Read this way, every imaging inverse problem has exactly three ingredients: a forward model (the operator $A$ in the case of linear imaging, how a scene turns into an expected measurement), a noise model (the statistics of $\mathbf{n}$, how that measurement is corrupted), and a prior (which scenes are plausible before we look at all). The first two together are the likelihood $p(\mathbf{y}\mid\mathbf{x})$; the third is $p(\mathbf{x})$. Naming the three is more than tidiness. It is what lets you change one without disturbing the others: a new camera may change both the operator and the noise; a new image model changes the prior. This is one appeal of the Bayesian perspective, it is mathematically modular (although actually solving the problem may require specialization).
The likelihood describes measurement variability for a fixed candidate scene. Under independent additive Gaussian noise of variance $\sigma^2$,
For covariance $\Sigma_n$, replace the squared norm by $(A\mathbf x-\mathbf y)^\top\Sigma_n^{-1}(A\mathbf x-\mathbf y)$. If the covariance depends on the candidate signal, its normalization also varies and must be retained. In photon counts, shot noise is Poisson, while read noise often dominates the deepest shadows. A Gaussian approximation becomes useful at larger counts. Saturation requires a censored likelihood: a clipped value represents a tail event, not an exact measurement at the threshold (Chapter 2.8, Noise, signal-to-noise ratio and dynamic range).
The likelihood says: if the true unknown (scene) were $\mathbf{x}$, then the imaging system would yield an expected measurement of $A\mathbf{x}$. This means that the noise value would need to be $\mathbf{n} = \mathbf{y}-A\mathbf{x}$ to result in our observation. Given a noise model (which can depend on $\mathbf{x}$, as with the signal-dependent shot noise of Chapter 2.8, Noise, signal-to-noise ratio and dynamic range), we can score the probability of $\mathbf{n}$.
The likelihood tells us how likely a measurement $\mathbf{y}$ would be if the true unknown were a given $\mathbf{x}$. The remaining question is how likely $\mathbf{x}$ itself is. The prior $p(\mathbf{x})$ is everything we know about the probability of $\mathbf{x}$ before looking at any particular measurement. A uniform prior requires a specified bounded domain; there is no proper uniform density over all real-valued images. But as we discussed in the section above, not all arrays of RGB values are equally likely to be natural images. The Bayesian perspective lets us combine this notion of prior cleanly with a forward and a noise model.
A density $p(\mathbf x)\propto\exp[-\lambda R(\mathbf x)]$ gives an optimization penalty through its negative log, provided it is normalizable on the stated domain. A gradient-only penalty does not constrain an arbitrary constant offset, so it needs an intensity constraint or an explicit treatment as an improper prior. A useful regularizer need not have been designed as a complete generative model.
Although we have developed it on the linear model $\mathbf{y}=A\mathbf{x}+\mathbf{n}$, nothing about the Bayesian view requires linearity. It needs only a forward model that predicts the expected measurement from a candidate scene, linear or not (a nonlinear sensor response, a rendering engine, a full light-transport simulation), and, usually, a noise model to score the mismatch. Bayes' rule then fuses that likelihood with the prior exactly as above. The linear case is special only in that a Gaussian prior and Gaussian noise give a Gaussian posterior and its MAP estimate collapses to the tidy regularized least squares of the linear-methods part; the framework itself is general, which is why the same three ingredients organize everything from classical restoration to modern generative models.
Synthetic training pairs exploit the forward model: start with a reference image and simulate its degradation. The reference is ground truth for that simulation, not automatically for a real camera. A processed RGB image already contains noise and ISP artifacts, and cannot fully specify depth-dependent blur or newly visible surfaces. Chapter 7.2, Machine learning develops these data choices.
There is a useful reversal of the usual labeling problem in machine learning. For recognition, a photograph is an input and a person may have to supply its label. For restoration, a suitable reference photograph can serve as the target, while a forward model creates a degraded input. Blur the reference and add noise to teach deblurring; select color samples to create a synthetic mosaic. The same asymmetry that makes inversion difficult makes simulated pairs inexpensive. The limitation is the fidelity of the simulation and reference, not the availability of a way to run the forward operator.
An explicit prior is also a claim we can examine. A sparse-gradient model says something different about the world from a Gaussian smoothness model or a learned multimodal distribution. We can compare those claims with image statistics and study where they fail. This is one reason to interpret a penalty through its implied distribution when possible: a numerical preference becomes an assumption about which scenes the method favors. The interpretation does not make the assumption correct, but it gives us a concrete object to criticize and improve.
7.1.3 Optimization, inference, loss⧉
We now have everything the problem is made of: a forward model (linear or not), a noise model, and a prior; together, a posterior $p(\mathbf{x}\mid\mathbf{y})$ that scores every possible scene. But a posterior is a whole distribution over answers, and a user wants one image. So the question that every restoration method answers, whether or not it admits it, is: given the distribution of plausible solutions, what single answer do we return? There is a small menu, and the choice is a real one: it is the moment inference, optimization, and the notion of "loss" all enter.
Pick the most probable one: MAP. The simplest answer is the peak of the posterior, the scene at a maximum of the posterior density: the maximum-a-posteriori (MAP) estimate. Take the negative logarithm of the posterior and minimize:
For independent Gaussian noise and prior $\mathbf x\sim\mathcal N(\mu,\tau^2I)$, multiplying the negative log posterior by $2\sigma^2$ gives
Thus the regularization weight is $\lambda=\sigma^2/\tau^2$ under this convention. Non-Gaussian priors give nonquadratic penalties. For continuous variables, MAP maximizes a density, not the probability of an individual point; changing parameterization can change the mode.
With Gaussian measurement noise, a negative log prior supplies the regularizer in a least-squares objective. A Gaussian prior gives a quadratic penalty; sparse priors give other penalties. Check whether the proposed prior is normalizable and what image variations it suppresses.
Minimize an aggregate distance to the plausible solutions, and which distance? MAP returns the single likeliest scene, but that is not the only thing one might want. Instead of the peak, we can return the answer that is, on average, closest to the true scene, closest in some chosen notion of distance. Minimize the expected squared error and you get the MMSE estimate, the posterior mean, the average over all plausible scenes (and, a fact we lean on hard later, what an estimator optimal for squared error returns, the hinge to Chapter 7.5, Image priors and Tweedie's formula). But "closest" begs the question which distance? Squared pixel error is merely the convenient choice (its derivative is linear, so it is easy to optimize) not the right one. Two images a person would call identical can be far apart in pixel $L_2$, and two that $L_2$ calls close can look obviously different. The distance you minimize is a loss, and choosing it is choosing what you are willing to be wrong about: a numerical loss ($L_2$, $L_1$) is tractable but perceptually naive; a perceptual loss (SSIM, LPIPS, a deep feature distance) is closer to what the eye scores but harder to optimize and easier to game (→ Chapter 9.3, Image metrics; perceptual and adversarial losses in Chapter 7.3, Deep learning). The loss defines what the method treats as a good reconstruction.
For a chosen loss $\ell$, the Bayes estimate minimizes posterior expected loss:
Squared error gives $\hat x=\mathbb E[x\mid y]$; scalar absolute error gives a posterior median. A posterior sample instead draws one possible $x$ according to $p(x\mid y)$. Sampling can preserve variation that averaging removes, but it does not identify the actual scene. Approximate generative restoration methods must also be checked for whether they sample the claimed posterior.
For a scalar measurement $y=x+n$, take $x\sim\mathcal N(0,1)$ and $n\sim\mathcal N(0,1)$ independently. The posterior is $\mathcal N(y/2,1/2)$. If $y=2$, both MAP and MMSE are 1; samples vary around 1. The uncertainty remains even though the two point estimators agree.
The estimators answer different questions. MAP selects a density maximum, MMSE minimizes expected squared error, and sampling represents alternatives under the posterior. Non-Gaussian posteriors can separate them substantially, but need not do so. A decision loss specifies the cost of an estimate; a likelihood describes how measurements arise. Neither should be confused with the image prior Kay, Estimation Theory.
Two practical observations close the Bayesian picture. The first is good news: in photography, data is relatively easy to obtain. Building or learning a good prior $p(\mathbf{x})$ wants a great many example images, and photography has them by the billion, unlike, say, black-hole imaging, where the Event Horizon Telescope must reconstruct a picture from a single, precious, radically incomplete measurement with essentially no ground truth to lean on. By comparison we are spoiled for data. The caveat is that fully ground-truthed data is rarer than it first appears: even for demosaicking, the "true" full-color image hiding behind a Bayer mosaic is not something a camera can simply hand you, so the data we train and validate on is often itself a stand-in for a ground truth we never quite had.
The second observation is that the prior is there whether you name it or not. A bilateral filter, an edge-aware demosaicking rule, a hand-tuned denoiser, an end-to-end neural network. None of these mentions a prior, yet each embeds one: a bet about what natural images look like, welded into the method's machinery. The bet gets made regardless; the only question is whether you made it on purpose. This is the quiet virtue of the Bayesian formulation: it cleanly disentangles the three ingredients (forward model, noise, prior) from the inference method that fuses them, which is more modular and theoretically cleaner (sharpen the prior, and every task that borrows it improves at once; → Chapter 7.5, Image priors). That clean separation is not always achievable, and insisting on it can leave performance on the table: a tightly coupled end-to-end method that refuses to factor may simply work better. But knowing where your prior lives, even when it is implicit, is much of the difference between engineering and luck. So the natural next question is the one the rest of the chapter answers: what is that prior?
Before turning to how modern methods learn that prior, step back and weigh what this Bayesian framing buys and where it strains. Its strength is organization: everything is written in terms of the forward model, and measurement, likelihood, noise, and prior each occupy a named, separate slot, fused by one clean rule instead of an ad-hoc pipeline; sharpen any single piece and the rest carry on unchanged. Its limits are the flip side of that same tidiness, and they are the three we keep bumping into. First, the forward model: the whole construction assumes we know $A$ and the noise, and often we do not, or know them only approximately (the model-error worry of Chapter 6.7, Image formation as a linear system). Second, the prior: a formula for $p(\mathbf{x})$ is easy to write down and very hard to make good, and hand-built priors capture only the crudest image statistics. Third, the inference: for anything richer than a Gaussian, actually computing the MAP estimate or the posterior mean is costly and sometimes intractable. Much of what follows is a direct response to these three limits, and the response that has taken over the field is to stop deriving the prior and the inference by hand and start learning them from data.
Suppose the data permit two distinct edge positions. Averaging the corresponding sharp images can produce a softened or doubled edge that is not itself a likely scene. Reporting one posterior mode preserves one interpretation; sampling can expose both possibilities over repeated runs. The conditional mean can nevertheless minimize expected squared pixel error. There is no contradiction: the estimators answer different questions. Decide whether the output should minimize a specified loss, show one plausible explanation, or communicate uncertainty before choosing how to summarize the posterior.
7.1.4 Deep learning, amortized inference, and generative AI⧉
Everything so far has kept the three ingredients apart and fused them by hand: write down a forward model, a noise model, and a prior, then run an optimizer to produce an estimate. Deep learning collapses that pipeline into one step. Given enough matched input-output pairs, a network can be trained to map a degraded measurement straight to a restored image, learning, implicitly and at once, both the prior over natural scenes and the inference procedure that applies it. There is no separate regularizer to design and no optimization to run at test time; the whole computation is amortized into the network's weights during training, so restoring a new image is a single fast forward pass.
Training can use simulated pairs or carefully aligned captures. In both cases, model mismatch matters: a network trained with one blur or noise model can fail on another. A direct predictor amortizes an estimation procedure, but does not automatically enforce a known data constraint. Some methods retain explicit operator steps or optimization at inference time.
A generative model represents a distribution from which we can sample. Restoration additionally requires conditioning on the actual measurement. Rich prior knowledge does not make the camera model trivial: miscalibration, saturation, motion, and noise still constrain which outputs are credible. Chapter 7.4, Generative AI and diffusion describes this conditioning problem.
Learning often moves the modeling effort rather than removing it. A likelihood-based method needs a description of how the camera corrupts the scene. A predictor trained on simulated pairs needs that description in the simulator. A predictor trained on captured pairs needs acquisition and alignment that make the target meaningful. A Gaussian-blur simulator does not become a realistic lens model merely because its inverse is a neural network. The forward model remains part of the method even when it is no longer visible in the inference code.
7.1.5 When you can design the operator, and a teaser of information theory⧉
So far we have taken the operator $A$ as a given (handed to us by the optics) and asked only how to invert it. But a recurring theme of computational photography is that $A$ is often ours to choose. A coded aperture swaps the clear lens stop for a designed mask; a light-field camera inserts a microlens array; a coded exposure flutters the shutter; an end-to-end-optimized lens is allowed to shape its own point-spread function. The moment you have that freedom the problem changes character: it is no longer "invert this $A$," but "design the $A$ (together with its reconstruction) that makes the whole pipeline work best." Acquisition and recovery become a single, jointly optimized system, and the goal is to pick the operator whose measurements carry the most recoverable information about the scene, not merely the prettiest raw image. (Coded imaging redesigns $A$ to be better-conditioned; light fields and multi-shot capture make $A$ taller, both are this same move, and both get their own parts: Part 33, Computational optics and coded imaging, Part 31, Light fields and plenoptic cameras.)
Chapter 7.6, Information theory supplies one way to compare measurement designs: mutual information quantifies how much observing the data reduces uncertainty under a specified scene and noise model. Channel capacity additionally optimizes the input distribution under constraints. Neither replaces a task-specific reconstruction criterion. Compare designs at matched light, exposure, sampling, and hardware budgets; coding can improve conditioning while losing photons.
7.1.6 Matching the estimate to the task⧉
For restoration, state the forward model and likelihood, the source of prior knowledge, and the estimator or loss. Then check residuals in the measurement domain as well as the appearance of the output. A plausible reconstruction can disagree with the data; a data-consistent reconstruction can contain incorrect detail in the operator’s null space.
Denoising makes this distinction especially clear. Its likelihood favors candidates close to the observed pixels, while the prior describes which of those candidates are plausible images. Both contribute to the answer. Learned reconstruction and posterior-sampling methods change how this combination is computed; they do not remove uncertainty [@kadkhodaie-simoncelli-2021; @chung-etal-2023-dps; @kawar-etal-2022-ddrm].
7.1.7 Harder inverse problems: factorization⧉
Everything above assumed the forward model is linear: a known operator $A$ acting on one unknown. A whole second family of problems drops that, and because it recurs constantly it deserves a name before we leave the chapter. In these the measurement is a product of two unknowns, and we must recover both factors from their product alone.
The canonical version is pure arithmetic. Hand someone the number 12 and ask which two numbers were multiplied to make it. The answer is radically underdetermined: $3\times4$, $2\times6$, $1\times12$, $2.4\times5$ all fit, and nothing in the product alone prefers one. Only extra knowledge (a prior on what each factor is allowed to be) breaks the tie. That is the same logical move as regularization above, now doing even heavier lifting.
Factorization problems seek to decompose a number, or an image, into a product of two unknown pieces (scalars, or whole matrices) given only the product. This is fundamentally ill-posed: many factor pairs multiply to the same result, and unconstrained formulations often admit the degenerate "put everything in one factor, leave the other the identity" split. Additional constraints, calibration, or priors breaks the tie. In the fully general case it is our linear problem $\mathbf{y}=A\mathbf{x}+\mathbf{n}$, except we must now solve for the operator $A$ as well as the unknown $\mathbf{x}$. Examples include white balance, intrinsic images, matting, and blind deblurring.
In the fully general case, factorization is our familiar linear problem $\mathbf{y}=A\mathbf{x}+\mathbf{n}$ with the ground shifted beneath it: now $A$ is unknown too, and we must recover the operator and the scene it acted on from their product alone. That extra unknown is what tips these problems from merely ill-conditioned to ambiguous.
A striking number of imaging problems are exactly this shape:
- White balance: the raw sensor value is (to first order) the illuminant times the surface reflectance: $\text{observed}=\text{light}\times\text{material}$. From the product we must separate what color the light was from what color the object is. How hard the factorization is depends on how much we let the light vary. Global white balance assumes one illuminant for the whole frame, so a single triplet of coefficients (three numbers) explains every pixel: there are fewer illumination parameters, but unknown reflectances still leave an ambiguity. Local (spatially varying) white balance lets the light change across the scene, so each pixel becomes its own independent product of two color triplets, the local light times the local material. That is far more ambiguous, and it takes a strong prior on how illumination varies to pin it down (Figure 4.2.3) [@hsu-etal-2008; @boyadzhiev-etal-2012].
- Intrinsic images: an image is shading times reflectance (albedo), $I=S\times R$: the geometry-and-illumination term times the material term. Split them and you can relight, recolor, or lift shadows.
- Matting: a composited pixel is $\alpha F+(1-\alpha)B$: foreground color, background color, and opacity $\alpha$, coupled multiplicatively into one observed color (the equation of Compositing, segmentation and matting).
- Blind deblurring: the blurry photo is the sharp image multiplied by an unknown blur matrix (blurring is a linear operator, so the observation is the product of that operator with the image) and we know neither the image nor the blur (Chapter 16.4, Blind deblurring).
These factorization (or blind, or separation) problems share two features. First, they are even more ill-posed than a linear inverse problem, because an unconstrained model can admit a trivial split: dump everything into one factor and leave the other the identity (the "$1\times12$" answer, which for blind deblurring is precisely the "no blur, the image is already sharp" trap). Additional constraints, calibration, or priors (illuminants are few and smooth, reflectances piecewise-constant, natural images sparse-gradiented, mattes nearly binary) can reduce ambiguity without guaranteeing uniqueness. Second, the forward model is no longer linear, but it is usually _multilinear_: linear in each factor separately while the others are held fixed ($\alpha F$ is linear in $F$ for fixed $\alpha$, and linear in $\alpha$ for fixed $F$; $S\times R$ likewise). That structure is exactly what keeps them tractable: it invites alternating schemes (freeze one factor, solve the other as an ordinary linear problem, alternate), which is why so many of these methods read as coordinate descent. Seen this way, white balance, intrinsic images, matting, and blind deblurring are the same structural problem in four different applications: recover the factors from their product, and use a prior to choose among the many that fit.
Recap: big lessons of this chapter
Natural images occupy only a tiny fraction of the space of all arrays of RGB values.
If we can computationally model the manifold of natural images, we can use it to exclude unlikely solutions to inverse problems.
A bilateral filter, an edge-directed demosaicker, and an end-to-end network each bet on what natural images look like. One states the bet as a penalty, one hides it in the machinery, one absorbs it from data, and none of them can avoid making it, because the measurement alone does not determine the answer. The only choice is whether the bet is deliberate and inspectable. Faced with any method in the chapters ahead, locate its prior first: it tells you which images it will quietly fail on.
With Gaussian measurement noise, a negative log prior supplies the regularizer in a least-squares objective. A Gaussian prior gives a quadratic penalty; sparse priors give other penalties. Check whether the proposed prior is normalizable and what image variations it suppresses.
Factorization problems seek to decompose a number, or an image, into a product of two unknown pieces (scalars, or whole matrices) given only the product. This is fundamentally ill-posed: many factor pairs multiply to the same result, and unconstrained formulations often admit the degenerate "put everything in one factor, leave the other the identity" split. Additional constraints, calibration, or priors breaks the tie. In the fully general case it is our linear problem $\mathbf{y}=A\mathbf{x}+\mathbf{n}$, except we must now solve for the operator $A$ as well as the unknown $\mathbf{x}$. Examples include white balance, intrinsic images, matting, and blind deblurring.