6.1 Introduction to forward and inverse linear methods🔗⧉

A blurred edge spreads a sharp transition across several pixels. If we know how that spreading happened, can we recover the edge? A short row of samples is enough to show both the calculation and its limits.

The forward problem predicts a measurement from a scene and a camera model. The inverse problem estimates the scene from the measurement. We will use a known linear model; learning an unknown blur or geometry is an additional estimation problem. Deblurring, denoising, demosaicking, and super-resolution differ in what was measured and what assumptions recovery needs.

We compare a blur, an idealized corner camera, and a sampling grid. A blur attenuates detail and may erase some patterns entirely. A corner camera records a weak indirect signal. A coarse sampling grid cannot distinguish arbitrary fine patterns. These examples separate noise sensitivity from non-uniqueness.

6.1.1 A one-dimensional blur, forward🔗⧉

Strip the image down to a single row of pixels, a one-dimensional signal $x = (x_1, x_2, \dots, x_n)$. Think of it as one scanline of a photo, or a flat 1-D world. Now blur it in the gentlest possible way: replace every sample by the average of itself and its two immediate neighbors. This is the box blur of radius one,

Each output sample is a little local average, so sharp features soften and fine wiggles wash out, which is exactly what optical blur does to a real photo (the full two-dimensional story is Chapter 6.2, Convolution).

Now write the same operation as a single matrix multiply. Stack the outputs into a vector $y$, collect the averaging weights into a matrix $A$ with one row per output sample, and the blur becomes

For a five-sample signal, with the edges handled by repeating the end pixel, the matrix is small enough to print in full:

Row three gives $y_3=(x_2+x_3+x_4)/3$. The repeated interior weights make a narrow band, while the first and last rows encode the chosen boundary rule. Other cameras need not have this banded structure.

This is the forward model: hand it a scene $x$, and it returns the measurement $y$. Computing it is trivial, a few multiplies and adds per sample, or in code a single loop:

# forward: 1-D box blur of radius 1, edges replicate
for i in range(n):
    y[i] = (x[clamp(i-1, 0, n-1)] + x[i] + x[clamp(i+1, 0, n-1)]) / 3

6.1.2 Running the blur backward: the idealized case🔗⧉

Now the interesting direction. Suppose someone hands you only the blurred signal $y$ and tells you it came from a box blur. Can you recover the sharp original $x$? This is deblurring, and the answer is that it is nothing but linear algebra. You have a linear system, $y = A x$, in which $A$ and $y$ are known and $x$ is the unknown, and solving a linear system is exactly what linear algebra is for. When $A$ is invertible you simply write

which in code is one call to a linear-system solver:

# inverse: recover the sharp signal by solving the linear system y = A x
x_hat = solve(A, y)     # solve the system without explicitly forming the inverse

This five-sample matrix is invertible. With exact measurements and arithmetic, solving recovers the original. That is a property of this matrix, including its size and boundary rule; a different box-blur matrix can be singular.

Write the two matrices side by side and they look nothing alike. The forward blur $A$ is the banded local average from before; its inverse is a full matrix, in which every recovered sample draws on all five measurements:

Undoing a local average is inherently non-local: $A^{-1}$ reaches across the whole signal, with alternating signs and larger weights. That growth is the first hint of the trouble the next section runs into, where those larger weights multiply not only the signal but any noise riding on it.

6.1.3 The catch: noise and ill-conditioning🔗⧉

If it were always this easy the rest of the part would be short. Unfortunately, a real measurement is never exactly $Ax$; the sensor adds a little random noise (see Chapter 2.8, Noise, signal-to-noise ratio and dynamic range), so what you actually hold is

Invert that and you get

The error is the measurement noise transformed by the inverse. A pattern strongly attenuated by the forward operator needs a large inverse gain, which also amplifies noise in that direction. A pattern mapped exactly to zero cannot be recovered uniquely. Blur width alone does not determine the difficulty: the kernel, sampling, and boundary rule all matter.

A practical reconstruction balances agreement with the measurement against assumptions about the scene. Regularization suppresses unstable components at the cost of bias. Chapter 6.7, Image formation as a linear system develops that tradeoff; Chapter 6.9, Efficient linear solvers addresses the computation. An accurate solver cannot repair an incorrect forward model.

6.1.4 A corner camera: reading the edge of a shadow🔗⧉

Let's look at a more surprising inverse imaging system. Consider standing in a room near the corner of a wall, where part of the scene is hidden from you around the edge (Figure 6.1.5⛶ fullscreen). You cannot see the hidden region directly. But light from it spills onto the floor near the corner, and the wall's edge lays a soft penumbra there, a gradual band from shadow to light rather than a crisp line. That penumbra, almost too faint to notice, is a photograph of the hidden region in disguise, and it can be read out (Bouman et al., Turning Corners into Cameras, 2017).

For a simple discrete model, suppose successive floor samples reveal one more angular bin of the hidden scene, with equal known weights and no background. Each measurement is then a cumulative sum. Its matrix $A$ is lower triangular, and $y=Ax$. This model illustrates the role of the occluding edge; the real measurement also depends on light transport, floor reflectance, background illumination, and noise.

Both matrices are as simple as the story. Written out for a five-angle scene, the forward operator (the cumulative sum) is a lower-triangular block of ones, and its inverse is the differencing matrix, with $1$ on the diagonal and $-1$ just below it:

Each row of $A$ sums the hidden scene up to its angle; each row of $A^{-1}$ takes a single difference.

With a known zero baseline $y_0=0$, neighboring differences recover the angular-bin contributions in this ideal model:

# corner camera: the penumbra p is a cumulative sum of the hidden scene by angle;
# difference it to recover the scene
scene[0] = p[0]         # known zero baseline before the first bin
for i in range(1, n):
    scene[i] = p[i] - p[i-1]

The difference extracts what the next floor measurement adds. Bouman et al. use video variations in the penumbra to reconstruct one-dimensional angular information about moving hidden objects Bouman et al. 2017 (corner camera). The cumulative-sum example does not recover a complete hidden photograph from an arbitrary floor image. See Chapter 38.1, Accidental cameras for related occluder-based systems.

Differencing is simple to compute, but it does not remove noise. For independent measurement errors of variance $\sigma^2$, an interior difference has variance $2\sigma^2$. A faint change can remain difficult to detect against a bright background. Fine angular sampling also changes the integration weights; restoring radiance per unit angle requires accounting for those bin widths.

Imagine ordering the hidden scene into angular slivers behind the corner. As the observation point moves along the visible floor, the wall edge reveals another sliver. In the simplified model, one floor sample therefore contains the contributions already visible to the preceding sample plus one new contribution. Subtracting the neighboring measurements cancels the shared contributions and isolates the newly revealed sliver. This is why a cumulative-sum matrix and a difference operator appear in the model. Unequal transport weights, varying floor reflectance, background light, and noise make the real subtraction less clean, but the edge is doing useful optical coding before any computation begins.

6.1.5 A different challenge: sampling and aliasing🔗⧉

So far the scene has arrived already discretized, a row of samples we could stack into a vector. But turning a continuous world into that row of numbers is a challenging step because it loses information. Reading a continuous signal at evenly spaced points is called sampling. Creating a thumbnail also involves sampling, now of a reconstructed discrete image. For fixed weights it is linear. Reducing the number of samples gives a matrix with fewer rows than columns, so arbitrary input values cannot be recovered uniquely.

Undersampling does something worse than lose information: it disguises it. A pattern that varies faster than the samples can follow does not simply vanish; it comes back as a slower pattern that was never in the scene. The moiré shimmer on a striped shirt on video, the picket fence that strobes past a train window, the wagon wheel that spins backward in an old film: all the same effect, called aliasing. In one dimension it is stark. Sample a fast sinusoid at too few points and the samples land exactly where a much slower sinusoid would put them, and nothing in the measurement tells the two apart (Figure 6.4.1⛶ fullscreen). The fast signal has taken on the alias of a slow one. The same masquerade plays out in two dimensions on a real photograph, where a fine, regular texture folds into bold moiré bands (Figure 6.4.2⛶ fullscreen).

An undersampled sinusoid and its alias produce identical samples. The measurement alone cannot tell them apart. A prior may favor one explanation, and additional shifted frames may distinguish them; neither changes the non-uniqueness of the single unconstrained measurement. The same distinction applies to any null space, including one created by blur.

Low-pass filtering before sampling suppresses frequencies the output grid cannot represent. It trades fine detail for fewer false patterns. Practical filters leave some residual aliasing, whereas the sampling theorem assumes an exactly band-limited signal. Chapter 6.4, Sampling, downsampling, and aliasing and Chapter 6.5, Resampling and upsampling develop the model and its implementation.

6.1.6 What the measurement determines🔗⧉

Before choosing a reconstruction method, specify the forward operator, its boundary rule, the noise model, and any additional measurements. Then ask whether different inputs can produce the same output and whether small measurement errors can produce large reconstruction errors.

Chapter 6.7, Image formation as a linear system develops these questions for general operators. Chapter 6.9, Efficient linear solvers separates the recovery objective from the numerical method used to solve it.