BookNotesSlidesP-sets Syllabusdraft · v0.2.49.1
◀▲⌂▶
💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.

33.0 COMPUTATIONAL OPTICS AND CODED IMAGING⧉

Two lessons have run through everything so far, and this part is built on top of both. The first is image formation: a sensor pixel does not see a point of the world, it sees a cone of light rays, and it reports their integral, one number that sums everything that crossed the aperture and landed there. The second is ray selection: the optics decides which rays land on which pixel. A pinhole selects one thin pencil per pixel; a lens gathers a whole cone and converges it; focus is just the name for getting that selection right. Put the two together and a camera is a machine that, pixel by pixel, picks a set of rays and adds them up.

The constraint is simple. In the incoherent radiance model used here, fixed passive optics form linear, nonnegative combinations of incoming radiance. It can gather light and sum it, weight some directions more than others, even throw some away, but it cannot subtract one ray from another, and it cannot make a pixel a non-linear function of its rays. Glass only adds light; it does not negate it. The linear sensor signal is therefore a weighted sum with nonnegative weights, not necessarily normalized to an average, and that single restriction bounds what optics alone can do.

Coded imaging is the idea of letting optics and computation together decide the ray combination behind each final pixel. The move is to split image formation into two stages. The optics no longer has to make the picture; it only has to encode the scene's rays into an intermediate measurement, a code that is not yet a photograph, often a deliberately scrambled or blurred image. Then computation decodes that measurement into the final image. The name is borrowed from communication theory on purpose: the optics is an encoder, the sensor records the code, and the algorithm is the decoder that turns it back into a picture. We are after the same goal as a good communication code: capture the information now and reconstruct it faithfully later.

fig-coded-imaging-pipeline
Figure 33.0.1. Coded imaging splits image formation into an optical encode and a computational decode. Top: a plain lens forms the final picture directly. The optics both selects and combines the rays into each pixel. Bottom: in coded imaging the generalized optics encode the scene's rays into an intermediate code on the sensor (a measurement, not yet a picture), and computation decodes that code into the final image. Because the computational half is free of optics' limits, it can weight rays negatively and combine them non-linearly. Re-creates the lecture's encode→decode diagram.

The reason to bother is that the decoder is not bound by the optics' limits. Computation can give rays negative weights. That is exactly what deblurring does when it subtracts the blurred neighbors back out of a pixel, and it can combine them non-linearly, applying thresholds, priors, or a learned reconstruction. The optics is stuck with linear, positive averages; the computation is not, and coded imaging is the discipline of designing the optical encoder and the computational decoder together so the pair captures and recovers more than either could alone.

fig-optics-linear-positive-vs-computation
Figure 33.0.2. Optics versus computation, as ray weights. Left: a lens forms each pixel as a linear, positive weighted sum of its rays, a blur kernel whose weights are all non-negative; a lens can add light but never subtract it. Right: computation is free to use negative weights (the center-positive, surround-negative stencil of sharpening / deconvolution) and non-linear combinations (thresholds, priors, learning). That extra freedom is what the decode half of coded imaging buys.

Two goals organize the whole part. The first is better end-to-end capture of information, and with it resilience to noise. If the optics encodes the scene with a blur that is invertible, one that retains useful contrast over a specified frequency band, then regularized reconstruction can recover useful detail over a greater operating range. Invertibility alone does not guarantee stable recovery in noise. The encoder's job is to choose a code that survives the trip through the sensor (and its noise) with the information intact; the decoder's job is to invert it. The second goal is choosing imaging parameters after the shutter has closed. If the intermediate code captures more than any one picture needs, the computation can form many pictures from it, and decide the parameters a posteriori, most famously the focus, recomputed at will from a single capture (the refocusing of Part 31, Light fields and plenoptic cameras), but the same logic reaches aperture, viewpoint, and more.

The part runs from the canonical example outward to the general machinery and the hardware. Chapter 33.1, Wavefront coding is the textbook first instance of the encode→decode story: a cubic phase plate in the pupil encodes the scene with a blur that varies little over a designed defocus range, and one fixed deconvolution decodes it into an image with a greatly extended depth of field, from one capture, subject to its depth range, sampling, and noise limits. Chapter 33.2, Compressive sensing is the purest statement of the thesis: the sensor takes far fewer measurements than there are unknowns, each a coded linear combination of the scene, and computation recovers the whole signal by exploiting sparsity (the single-pixel camera is its emblem). Chapter 33.3, Coded aperture puts a patterned amplitude mask in the lens, encoding depth into the blur so it can be read back out and inverted. Optical modulators, such as DMDs and liquid-crystal panels, are the programmable hardware that does the coding on the fly, the active cousins of the fixed masks. And Exotic / advanced optics pushes the idea to its limit, replacing the lens entirely with a mask or diffuser and reconstructing the image purely by computation. A closing chapter collects exotic optics, including GRIN lenses, metalenses, and non-linear optics, where the wavefront is shaped in other ways.

One sentence summarizes the part: a conventional camera forms the picture in glass, with linear positive ray sums; coded imaging lets the glass merely encode the rays and hands the picture to computation, which can use signed and nonlinear reconstruction while remaining limited by the information and noise in the measurements. Wavefront coding is where we start, because it is the cleanest place to watch a deliberate blur become a feature rather than a flaw.

Contents of this part

▸ full collapsible outline of this part