1.1 From digital to computational photography🔗⧉

A phone can combine several noisy exposures into one clear photograph, or use depth to blur the background behind a face. These pictures depend on both a physical measurement and a computation. Digital photography made that combination possible by recording the image as numbers; computational photography designs what we measure and how we turn those measurements into a picture.

Consider a dark room with a sunlit window. Exposures chosen for the room and for the view outside can be combined into a high-dynamic-range (HDR) image that retains both. Other methods estimate camera shake to reduce blur, stitch overlapping frames into a panorama, or record light from multiple viewpoints so we can refocus after capture. Each method uses additional measurements or assumptions. None can recover arbitrary information that the capture failed to record.

Even the most ordinary snapshot already depends on computation. As we'll see shortly, a typical sensor measures only one color channel at each pixel (red, or green, or blue), and the other two are reconstructed by an algorithm before you ever see a full-color image. Computation isn't an afterthought for fun special effects. It has become a fundamental aspect of image formation.

A phone has little room for a large aperture or sensor. Aligning and merging frames collects evidence over time, and denoising and tone reproduction help use that evidence. The small optics still matter: blur, flare, clipping, and motion can destroy information before processing begins. Computation and good hardware complement each other.

Generative models add another source of information: patterns learned from training images. They can synthesize or modify a picture, including details that the camera did not resolve. This makes restoration more flexible but raises a question that will recur throughout the book: which details are supported by the measurement, and which are plausible inventions? Borges's imagined library of every possible book offers an analogy for this breadth, but a learned model does not contain every possible photograph Borges 1941 Bottou & Schölkopf 2023.

1.1.1 Computation is the new optics🔗⧉

In a conventional camera, the lens forms an optical image and film or a sensor records it. Better lenses, emulsions, sensors, and exposure control improve that measurement. Processing has always influenced the final photograph, including through chemical development and printing; computation greatly expands the operations we can apply.

Computational photography allows the final picture to differ substantially from the optical image. We can combine exposures, estimate missing values, and use measurements of depth or ray direction. The attainable result depends on the complete system: capture, reconstruction, and display.

The optical image need not look like the desired result. It can instead be designed to preserve information that an algorithm can recover. This is optics and computation co-design: choose the measurement and reconstruction together. A coded measurement is useful only if the desired information survives with enough signal relative to noise.

Consider extending depth of field. Wavefront coding Dowski & Cathey 1995 and the lattice-focal lens Levin et al. 2009 (4D frequency / lattice-focal) deliberately blur the optical image over a range of depths. They aim to make the blur more consistent and easier to invert than ordinary defocus, so one reconstruction can sharpen objects at different distances (Figure 1.1.3⛶ fullscreen). In the usual incoherent imaging model, the sensor receives nonnegative sums of intensities; computation can also subtract and apply nonlinear operations. Deconvolution uses that freedom, at the cost of noise amplification and possible artifacts. The useful depth range and recoverable detail remain finite. See Part 33, Computational optics and coded imaging.

“Computation is the new optics” describes this expansion of image formation: we can improve the picture by changing the computation as well as the lens.

1.1.2 The goals of computational photography🔗⧉

The methods in this book serve several photographic goals:

• Prettier pictures: straightforwardly better-looking results: cleaner, sharper, better-toned, more pleasing color.
• Alleviate physical limitations: reduce the effects of hardware constraints: too little light, too little dynamic range, a lens too small or too shaky.
• Record more information: capture what an ordinary photograph discards: depth, scene structure, separate layers of the scene.
• More control after capture: give the photographer more control after capture rather than locking in every decision at the moment of the shot.
• New visual experiences: make images that simply have no single-shot equivalent, like a refocusable photograph or a navigable panorama.
• Reveal the invisible: surface what the eye cannot see on its own, from tiny motions and color changes to information outside the visible band.

Prettier pictures. A raw file needs a rendering before it becomes a viewable photograph: white balance, color conversion, and a mapping to the display. A neutral rendering and a photographer's preferred rendering can look quite different (Figure 1.1.4⛶ fullscreen). There is no single appearance intrinsic to the raw numbers.

Alleviate physical limitations. A phone can merge a burst of short exposures, preserving highlights in each frame while reducing noise in the combined shadows. The additional frames collect more photons; alignment and merging make their measurements useful (Figure 17.6.6⛶ fullscreen). Motion limits how much can be combined.

Record more information. Overlapping photographs can support a three-dimensional reconstruction and views from new camera positions (Figure 36.7.7⛶ fullscreen). Coverage, texture, and scene motion determine which parts can be reconstructed reliably.

More control after capture. Recording ray direction as well as position allows refocusing within the range supported by the capture (Figure 31.4.2⛶ fullscreen).

New visual experiences. Some images have no single-shot equivalent at all: you navigate them rather than look at them (Figure 20.4.5⛶ fullscreen).

Reveal the invisible. The camera had already recorded a face flushing with its pulse and a building swaying. What was missing was the processing that makes such changes visible.

A limitation can also be an expressive choice (Figure 1.1.5⛶ fullscreen). Leaving deep shadows black may make the lit shapes easier to read than opening every shadow. Freezing a fraction of a second lets us study a gesture that passed too quickly to see. More recovered detail is not always a better photograph.

The intended use determines how we evaluate an image. A calibrated measurement can be assessed against a physical quantity, with uncertainty attached. A perceptual model predicts visibility or appearance under specified viewing conditions. A photographer may instead choose a rendering for its expressive effect.

These criteria can disagree. Smoothing a face may reduce pixel error while removing texture the photographer wants. Preserving every shadow detail can flatten the composition. Human vision has measurable regularities, including contrast sensitivity and color matching, but perceptual metrics remain models with limited ranges of validity. Choose the criterion before comparing methods.

Between physical measurement and personal preference lies perceptual vision, which concerns how an image actually looks to a human eye. You might expect this to be hopelessly subjective, but it is not. Several aspects of human vision can be modeled quantitatively: the contrast sensitivity function, color appearance, lightness and brightness, visual masking, perceived sharpness, and full image-quality metrics such as SSIM and visible-difference predictors. "How it looks" can be measured and even predicted, though it is not pure physics. Within a model’s range of validity, this lets us engineer for the eye: tone-mapping to fit the eye's response, throwing away color detail the eye won't miss, choosing where to spend bits or where a little smoothing will go unnoticed.

1.1.3 What makes a technique useful🔗⧉

Beyond the problem a technique solves, there are the engineering qualities that decide whether a method graduates from a paper to a product. A useful technique is fast, cheap enough to run at the scale and latency its use demands, ideally interactively or on the device. It is predictable, with graceful degradation: when its assumptions are violated it should fail gently and visibly, a little softer, a little noisier, rather than catastrophically, throwing bizarre artifacts that are worse than having done nothing. And it is controllable and tunable: it exposes a few meaningful knobs a photographer or engineer can actually turn, rather than being a black box with one fixed behavior. Methods that score well on all three tend to win, even against more accurate but brittle or opaque rivals.

Establish a correct reference implementation before optimizing. Fast approximations made bilateral and local Laplacian filtering practical (Chapter 10.1, Bilateral filtering, Chapter 10.4, Local Laplacian filters, Chapter 29.1, Algorithmic speedups). But speedups are not guaranteed: memory, energy, and latency can require a different algorithm or a reduced output resolution. Measure these constraints early, then check each optimization against the reference.

There is a reason to tolerate a slow first implementation. It tells us whether the idea produces the picture we want before approximations make its errors harder to diagnose. Bilateral and local Laplacian filtering illustrate the sequence: establish the useful operation, then find a faster way to compute or approximate it. A fast implementation of the wrong operation is not progress. Keep the reference available while optimizing, so speed and accuracy remain separate questions.

1.1.4 Hand-designed algorithms, learning, and generative AI🔗⧉

Hand-designed methods specify how to reconstruct an image. Learned restoration estimates that mapping from examples. Generative methods learn a distribution of images and can sample possible outputs, often conditioned on a capture. These approaches coexist: a camera may use a conventional geometric alignment stage, a learned denoiser, and a hand-designed tone curve.

The distinction does not by itself tell us how faithfully a result follows the data. A hand-designed prior can erase real texture; a learned model can be constrained by the measurement. Generative restoration is especially useful when many plausible images explain the input, but the selected detail need not be the detail that was actually there. Evaluation must reflect whether the task is measurement, photographic rendering, or synthesis.

Rich Sutton's The Bitter Lesson argues that general methods able to exploit growing computation tend to outperform elaborate hand-coded knowledge over time. Natural-language processing gives the standard example. For decades, parsing a sentence meant hand-building grammars that encoded linguists' knowledge of syntax; statistical parsers trained on corpora of annotated sentences overtook them, and neural models later did so with still less hand-designed structure. The linguistic knowledge did not stop being true. It stopped being the thing programmers wrote down directly, and moved into the data, the annotations and the choice of what to measure. Imaging offers an important place for physical knowledge within learned methods: we can model projection, blur, sensor noise, and the color-filter mosaic to generate training pairs or constrain reconstruction. Those pairs are exact for the simulator, not necessarily for a real camera. Calibration errors, clipping, motion, and unmodeled processing can make synthetic training data misleading. Chapter 7.2, Machine learning and Chapter 7.3, Deep learning develop these approaches.

This also changes an expectation that developed around photographs. Perspective construction, photographic capture, and documentary practice all helped establish the image as evidence about a particular scene. Photojournalism gave special importance to the idea of recording an actual instant. That ideal was never a complete description of image-making: staging, framing, and manipulation remained. Generative synthesis makes the distinction especially visible because a convincing image need not originate in a corresponding event. The question is what relationship to an event a viewer is entitled to expect.

The physical knowledge accumulated in imaging still has work to do. A hand-designed denoiser puts assumptions directly into its algorithm; a learned denoiser can inherit them through simulated noisy/clean pairs, its architecture, or a measurement constraint. Knowledge of the camera has moved into a different part of the design. The practical question is where that knowledge is most useful, and how closely the simulated measurements represent the real capture.

1.1.5 Does photography still matter in the age of generative AI?🔗⧉

A generated picture of a toddler cannot replace a photograph of your child's first steps. The photograph is valuable because the capture is connected to that child and that moment. This connection to a particular event is often called indexicality.

It is a connection, not a guarantee of truth. Framing, staging, editing, and reconstruction all affect what a photograph communicates. Scientific and documentary uses therefore need evidence about how the image was captured and processed (Chapter 41.2, Authentication and Provenance (C2PA)). A plausible-looking result is insufficient.

A phone also makes capture available when the moment occurs (Figure 1.1.8⛶ fullscreen). Generative editing changes what we can do with that capture; it does not remove the value of having been there.

Learned methods already perform restoration in camera pipelines, but learning and generation are not synonyms. A network trained to denoise a raw image need not sample a generative model. When generative tools add unsupported content, the appropriate use and disclosure depend on whether the photograph is a personal creation or a record offered as evidence.

1.1.6 A multidisciplinary field🔗⧉

Computational photography draws on optics and sensor engineering to form measurements, algorithms and machine learning to reconstruct images, and perception and photographic practice to judge the result. Performance engineering determines what can run on a device. Ethics matters when the processing changes how we represent a person or an event.

A few neighboring fields overlap with computational photography so heavily that it helps to say exactly how they relate and differ.

Computer vision is the closest. A useful way to carve up vision is by its three ingredients: 2-D images (the pixels themselves: filtering, features, matching), 3-D (the geometry behind the pixels: depth, shape, pose), and meaning (semantics: what objects and scenes the pixels depict). Computer vision ultimately cares most about the last two: it treats the image as a means to recover 3-D structure or to understand content. Computational photography inverts that emphasis: its product is the 2-D image, a better photograph. But it freely reaches into 3-D and meaning when they help the picture: depth to fake a shallow-focus background, segmentation to brighten a face and not the sky, recognition to pick the frame where everyone is smiling. The same tools, pointed at a different goal.

Computer graphics runs the arrow the other way, synthesizing images from models of geometry, materials, and light. Computational photography borrows its rendering and image-based-rendering machinery (and, increasingly, shares it outright: gradient-domain editing, light fields, differentiable and neural rendering all live in both). Image processing is the classical signal-processing bedrock the whole enterprise is built on: filtering, transforms, sampling, and compression. Models of capture help choose the processing. And optics supplies the physics of image formation, which computational photography increasingly co-designs with the computation rather than treating as fixed (coded apertures, lensless cameras, end-to-end optimized lenses).

Because the field straddles these neighbors, so does its literature. A great deal of computational photography is published at the major computer-vision conferences (CVPR, ICCV, and ECCV) and at the graphics venue, SIGGRAPH (and SIGGRAPH Asia / ACM Transactions on Graphics). ICCP, the IEEE International Conference on Computational Photography, is a venue dedicated to the field.

These disciplines appear together inside a single feature. Take a phone's night mode. It leans on physics (catching scarce photons), electrical engineering (the sensor that catches them), algorithms (aligning and merging a burst of frames), AI (learned denoising), perception (tone-mapping the result so it reads naturally), performance engineering (finishing within a short capture budget on a phone without excessive energy use), and ethics (how much to "beautify" a face, and whether to tell the user), all at once. Each discipline constrains the others: a more expensive reconstruction can require fewer input frames or a longer wait for the picture.

1.1.7 Beyond photography🔗⧉

The methods used on photographs also apply to other measurements. The shared approach is to specify what an instrument records and reconstruct the quantity we want from it.

Start with the ordinary camera pointed at extraordinary jobs. The instant a camera stops being aimed by a human enjoying the view and starts being aimed by a machine making a decision, the processing serves a different task. Robotics and self-driving cars lean on cameras (and their cousins, depth sensors and lidar) to perceive, localize, and navigate. The alignment, depth, and motion estimation we develop for panoramas and refocusing are exactly what a robot needs to know where it is and what is moving. Surveillance and security, industrial inspection (spotting a hairline crack or a misplaced solder joint on a production line, where consistent lighting and repeatable measurements help detect defects), agriculture, drones, document scanning, and biometrics all use computational imaging for practical tasks. The picture is no longer the product; it is an input to a decision.

Then push the optics past what a handheld lens can do. Astronomy is computational photography at the grandest scale: telescopes stack thousands of frames to beat noise (the same averaging that powers a phone's night mode), use adaptive optics to cancel the atmosphere's blur in real time, and combine separate dishes into one synthetic aperture, the trick that let the Event Horizon Telescope assemble an Earth-sized virtual lens to image a black hole. At the opposite scale, microscopy does the same: light-sheet, confocal, and computational microscopes reconstruct 3D cellular structure, and super-resolution techniques reach past the diffraction limit we will meet in the optics part. Both require designing the measurement and reconstruction together.

Change the wavelength and the field opens again. Visible light is a sliver of the electromagnetic spectrum, and imaging lives all along it. Infrared gives thermal cameras and night vision; ultraviolet reveals forgeries and surface detail; X-ray imaging, in the form of computed tomography (CT), reconstructs slices or volumes from X-ray attenuation measurements along many paths, a reconstruction problem; radio gives us radio astronomy and synthetic-aperture radar (SAR) that maps terrain through clouds and darkness. Much of medical imaging is this same discipline under a different name: CT, magnetic resonance imaging (MRI), positron-emission tomography (PET), and ultrasound are all reconstruction problems, recovering an image from indirect measurements using related reconstruction ideas, including the regularized inverse problems this book builds up to. A radiologist's scanner and a phone's night mode use closely related reconstruction ideas.

And the techniques do not even need an image at all. Sampling, filtering, the Fourier transform, denoising, priors, inverse problems. The toolkit is really a toolkit for signals, and a signal can be anything. Audio is the obvious neighbor: a sound is a one-dimensional signal, and the same spectral and statistical ideas drive noise reduction, compression, and synthesis there. One example shows the connection: Paul Lamere's Infinite Jukebox takes a song, finds beats that sound nearly identical, and quietly jumps between them to extend the track, with success depending on the available transitions, which is precisely the idea of Video Textures (Schödl et al. 2000), a computational-video technique that finds similar frames in a short clip and loops between them to synthesize a video of arbitrary duration with varied transitions, transplanted from pictures to sound.

Run the trade the other way, too: point a camera at a quantity that has nothing to do with light and let computation read it out. Two examples make the point. GelSight turns a camera into a sense of touch: press a slab of clear elastomer, with its outer face painted with an opaque membrane, against an object, and the gel molds to the surface's microscopic relief; a small camera behind the gel photographs that deformation under raking colored light and reconstructs a dense 3-D height map by photometric stereo. Its resolution depends on the optics, camera sampling, illumination, and mechanics of the membrane. It can read the ink of a banknote, the wear on a machined part, and the grain of a fingerprint, and it has become a workhorse for robotic manipulation, where a gripper needs to feel contact, slip, and texture, right down to vision-based tactile sensors built into the fingers of a dexterous robot hand Romero et al. 2024 (EyeSight Hand). Touch becomes an image that can be analyzed with image-processing tools Johnson & Adelson 2009. The second example points a camera at vibration: ordinary objects tremble by thousandths of a pixel in response to sound and force, and that motion, amplified and demodulated, turns a silent video into a measurement of the world's movement. A bag of chips behind soundproof glass becomes a microphone. The visual microphone recovers intelligible speech from the sub-pixel wobble of its surface Davis et al. 2014; the same sub-pixel motion, read as a modal signature, recovers an object's material properties Davis et al. 2015 and lets a single camera perform structural health monitoring of a bridge or a building, identifying its resonant frequencies and mode shapes from across the street Chen et al. 2015 Davis 2019. We return to both threads in Part 38, Revealing the invisible; these examples show how images can serve as measurements of other quantities. Touch and vibration are signals, and a camera plus computation is a remarkably general instrument for measuring them.

Many sensors make a quantity visible indirectly. Fluorescent markers emit light; an elastomer deforms under pressure; scintillators convert absorbed X-rays or gamma rays into visible light. A camera then turns that light into electrical measurements. In a semiconductor photosite, absorbed photons can generate electron–hole pairs whose charge is collected. This conversion, described in Chapter 2.7, Sensors: photosites, CCD vs CMOS, is one useful transduction mechanism among many, including the acoustic measurements used in ultrasound.

Part 39, Adjacent fields and applications develops these connections across imaging instruments and other signals.

The optical readout is often a design choice. We can arrange for pressure to deform a visible surface, for a chemical marker to fluoresce, or for an absorbed X-ray to produce a visible flash. Once the quantity modulates light, photodetection converts it into an electrical signal that familiar acquisition hardware can record. This is why an imaging course can teach useful measurement ideas far beyond cameras. We still have to model the conversion from the original quantity to light; linearity of the final sensor does not make every earlier conversion linear.