33.7 End-to-end optimization🔗⧉

Every optical code in this part was invented by a person. Dowski and Cathey derived the cubic phase plate; Raskar searched a handful of flutter sequences; Levin chose an aperture pattern and checked that its spectrum had no zeros. In each case a clever human picked the encoder, then a separate effort built a decoder to match it. That division of labor is exactly what end-to-end optimization dissolves. The idea is to write the whole imaging pipeline, including the optics, the sensor, and the reconstruction algorithm, as a single differentiable function, attach a loss to whatever you ultimately want out of it, and let gradient descent choose the optical design and the algorithm at once (Sitzmann et al. 2018). The lens stops being a fixed thing the algorithm must cope with and becomes part of the network being trained. People call the result deep optics, and it is the natural last chapter of a part whose every page has argued that building a camera is choosing the operator $A$: here, you stop choosing it by hand and let the optimizer choose it for you.

33.7.1 The pipeline as one differentiable graph🔗⧉

To optimize the optics with gradients you must first be able to differentiate through the camera. So you write image formation down as a forward model whose every step is differentiable, parameterized by the physical quantities you are allowed to change. The optical element (a refractive surface, a diffractive plate, a coded aperture, or a metasurface) is described by a small set of parameters $\theta$: a height map of the surface, or equivalently the phase profile it imposes on the wavefront across the pupil. From $\theta$, wave optics gives the point-spread function $\mathrm{PSF}(\theta)$, the image of a point of light, by a differentiable computation (a Fourier-optics propagation, the same machinery that produced the cubic plate's PSF in Chapter 33.1, Wavefront coding). The sensor then forms the captured measurement by convolving a locally shift-invariant scene patch with that PSF (or applying a spatially varying operator when depth or field position changes), adding noise, and clipping, a noisy forward model of the $\mathbf y = A\mathbf x + \mathbf n$ of Chapter 6.7, Image formation as a linear system, where now $A$ depends on $\theta$. Finally a reconstruction network with weights $\phi$ maps the raw sensor measurement to the desired output (a sharp image, a depth map, or a spectrum) and a task loss compares that output to ground truth.

Chain it together and the entire camera-plus-algorithm is one function,

read from the innermost operation outward as: take a scene $\mathbf x$, image it through optics $\theta$, capture it, reconstruct it with network $R_\phi$, and score the result against what you wanted. Clipping has zero derivative in saturated regions, and quantization is not ordinarily differentiable. Training therefore needs an explicit treatment of these operations, such as smooth surrogates or gradient estimators; sampled signal-dependent noise likewise needs care. Both $\theta$ (the optics) and $\phi$ (the algorithm) are knobs in the same graph, so a single backward pass computes the gradient of the final loss with respect to both (Figure 33.7.1⛶ fullscreen).

33.7.2 Backpropagating into the glass🔗⧉

The conceptual jump is small once the graph is built, but worth saying plainly because it is the whole chapter. Training a neural network means computing the gradient of a loss with respect to the network's weights and stepping downhill (Chapter 7.3, Deep learning). Here the "weights" include the optical parameters $\theta$, the height of the lens surface at every point. Because $\mathrm{PSF}(\theta)$ is differentiable, the chain rule does not stop at the sensor; it keeps going, back through the Fourier-optics propagation, and lands a gradient on the height map itself: which way should I deform the glass to lower the final error? Gradient descent answers that question and the next and the next, and the surface profile drifts, step by step, toward the shape that makes the paired reconstruction's job easiest. The lens is, quite literally, trained.

This is the precise sense in which end-to-end optimization is the ultimate "design $A$." The earlier chapters chose $A$ to be invertible by reasoning: invariance, flat spectra, and no zeros. End-to-end optimization chooses $A$ by gradient descent on the actual task, and it co-designs the inverse at the same time, so it does not even need $A$ to be invertible in any clean classical sense. It only needs the pair $(\theta,\phi)$ to work well together. The optimizer is free to discover codes no human would write down, because its only objective is the end result. It also optimizes for the whole task under the real nuisances (sensor noise, quantization, and the actual scene statistics of the training set) rather than for a tidy proxy like "flat MTF." The reconstruction is told what kind of mess to expect, and the optics is shaped to produce the most recoverable mess.

33.7.3 What gets designed: a height map, not a hyperparameter🔗⧉

It helps to be concrete about what comes out of training, because "learned optics" can sound more abstract than it is. The output is a physical surface: a height map you can fabricate. For a diffractive optical element or a metasurface, $\theta$ is the per-point height (or nanostructure) that sets the phase delay across the element; gradient descent returns a specific, often strange-looking, microstructured profile: a learned phase mask whose freeform pattern may differ from a textbook lens and produces the modeled PSF within fabrication tolerances (Sun et al. 2020). For a refractive element the same applies to a smooth surface profile, and for a coded aperture $\theta$ is the mask's transmission pattern, now learned rather than chosen by hand. This closes the loop on Chapter 33.3, Coded aperture. A physically feasible design must respect transmission, feature-size, thickness, dispersion, and fabrication constraints. After optimization, the trained $\theta$ can be handed to a fabrication process and a real element is etched or molded; the reconstruction network $\phi$ ships alongside it. The camera that results carries a piece of glass whose shape is a frozen answer to a gradient-descent problem.

33.7.4 A gallery of deep-optics results🔗⧉

The reason this matters is that, pointed at different task losses, the same end-to-end recipe designs cameras for jobs a conventional lens cannot do, and often does them better than hand-designed codes of the earlier chapters, because it tunes the code and the decoder jointly to the task (Figure 33.7.3⛶ fullscreen). A short gallery:

• Extended depth of field. Train the loss to demand a sharp all-in-focus image across a wide depth range and the optimizer learns a phase plate whose PSF is recoverable at every depth, an EDoF design in the spirit of Chapter 33.1, Wavefront coding, but with the mask and the deconvolver learned together rather than the cubic derived and the filter bolted on (Sitzmann et al. 2018; Chang & Wetzstein 2019).
• Monocular depth. Make the target a depth map, and gradient descent learns a PSF that encodes depth into the blur: a depth-dependent, asymmetric point spread the network reads back as range, so a single ordinary-looking shot yields depth, the learned successor to depth-from-coded-aperture (Chang & Wetzstein 2019).
• High dynamic range. Make the target a scene with highlights the sensor would normally clip, and the optimizer learns optics that spread saturating light into a recoverable PSF so the reconstruction can infer the clipped values. This yields a single-shot HDR camera designed end to end (Metzler et al. 2020).
• Hyperspectral and achromatic imaging. Make the target a full spectrum per pixel, or a sharp image across all wavelengths at once, and the learned diffractive element is shaped so that its wavelength-dependent PSF either encodes the spectrum for recovery or cancels chromatic aberration. This turns the diffractive element's usual curse, severe color dispersion, into a designed feature (Sun et al. 2020).

The part's move is the same, made sharp: in each case the optic is shaped to make the inverse problem easy for the network it is married to, and the network is shaped to invert that particular optic. Neither is designed in isolation; the pair is.

33.7.5 What it costs, and where it can go wrong🔗⧉

The accounting matters here as much as anywhere. The learned design is only as good as the forward model it was trained through: if the differentiable simulation of the optics, the sensor noise, or the fabrication tolerances is wrong, the optimizer happily exploits the gap and the real camera underperforms its simulation, the classic sim-to-real problem, sharpened because the "sim" includes physics. Fabrication itself is lossy: a height map is etched to finite precision, so the manufactured PSF differs from the trained one, and careful pipelines either model that error inside training or re-calibrate the measured PSF and fine-tune the reconstruction to it. The reconstruction network inherits all the cautions of Chapter 7.3, Deep learning. It can hallucinate plausible detail that was never measured, the standing "the prior fills in what the optics destroyed, and may invent it" caveat. And the joint loss landscape is non-convex in $(\theta,\phi)$, so the design you get depends on initialization (often seeded from a known good optic, like a cubic plate, so the optimizer refines rather than searches from nothing). None of this undoes the idea; it just means a learned camera is a trained system that must be validated like one, not a closed-form optic with a guarantee.

33.7.6 Where it sits🔗⧉

End-to-end optimization is less a new device than a new way to arrive at devices, and it reframes the whole part in hindsight. The cubic phase plate of Chapter 33.1, Wavefront coding, the flutter code of Chapter 33.5, Code in time (phase, amplitude), the aperture mask of Chapter 33.3, Coded aperture. Each is a point a human designer reached by reasoning, and each provides a baseline or initialization for joint optimization. Gradient descent does not guarantee that it will recover or improve on that design. The reason the optimizer can do this at all is the part's theory: a camera is a linear projection of the light field whose conditioning lives in its spectrum (Chapter 33.6, Theoretical analysis of imaging systems in the 4D light field Fourier domain), while a differentiable task loss supplies gradients for a chosen parameterization. End-to-end optimization simply lets that descent control both the optics and the algorithm at once.

It is also where this part shakes hands with Chapter 7.3, Deep learning. The reconstruction half is an ordinary network; the novelty is that the first layer is a piece of physics with learnable parameters, and that gradients flow into it. Pull on that thread and the whole stack (scene, optics, sensor, algorithm, task) becomes one trainable system, the most complete expression of the book's recurring claim that imaging is an operator you get to design. The earlier chapters showed you how to design $A$ by thinking. This one shows you how to let the gradient do the designing, and pair $A$, from the start, with the recovery that makes it pay off.