34.0 COMPUTATIONAL SENSORS: part introduction⧉
Most of this book takes the sensor as given, a grid of photosites integrating light into raw measurements from which RGB values are reconstructed, and asks what computation can do with the result. This part inverts the question: redesign the sensor itself. If a pixel value is an integral over the aperture, the pixel's area, wavelength, and time (Chapter 2.4, Image measurements as integrals over the plenoptic function), then a computational sensor is one that slices that integral differently, or measures something the conventional photosite throws away. The chapters survey the range: pixels that trade resolution for dynamic range or rearrange the color layout (Chapter 34.1, Assorted pixels, Chapter 34.2, Modern sensors (quad Bayer, in-sensor HDR, and beyond)); sensors that recover depth directly (Chapter 34.4, Depth sensors); detectors sensitive enough to count individual photons (Chapter 34.5, Single-photon sensors (SPAD, avalanche, photon counting)); sensors that read velocity by Doppler (Chapter 34.6, Doppler / velocity imaging), that fire only on change (Chapter 34.7, Event sensors) rather than on a clock, or that freeze ultra-high-speed events far beyond a normal frame rate (Chapter 34.10, Ultra High speed Imaging); plus specialized detectors and non-visual modalities (Chapter 34.8, Specialized and research sensors, Chapter 34.9, Extra sensors and non-visual data).
The sensor is a design variable, co-designed with the algorithm that reads it. Each chapter is the same move seen on the hardware side: change what is measured so that the computation downstream has more, or better-conditioned, information to work with. This is the choose the operator $A$ idea of Chapter 7.1, Imaging as Bayesian inference with probabilistic priors, pushed all the way down to silicon.
Contents of this part
- 34.1 Assorted pixels
- 34.2 Modern sensors (quad Bayer, in-sensor HDR, and beyond)
- 34.2.1 Quad Bayer, Tetracell, and nona-binning
- 34.2.2 In-sensor HDR: capturing range before the merge
- 34.2.3 Dual-pixel autofocus, on the same sensor
- 34.2.4 BSI and stacked sensors: compute under the pixels
- 34.2.5 The global-versus-rolling shutter trade
- 34.2.6 Beyond: nano-prism, organic, and event pixels
- 34.3 On-sensor HDR
- 34.3.1 Staggered / multiple-exposure readout (DOL-HDR)
- 34.3.2 Dual (and triple) conversion gain (DCG)
- 34.3.3 Split-pixel: a large and a small photodiode
- 34.3.4 Spatially-varying exposure (SVE) / assorted exposures
- 34.3.5 Lateral overflow integration capacitor (LOFIC)
- 34.3.6 Logarithmic, self-resetting, and counting pixels
- 34.3.7 On-sensor HDR versus multi-frame HDR
- 34.4 Depth sensors
- 34.5 Single-photon sensors (SPAD, avalanche, photon counting)
- 34.6 Doppler / velocity imaging
- 34.7 Event sensors
- 34.8 Specialized and research sensors
- 34.8.1 Geiger-mode avalanche-photodiode arrays: photon-counting laser radar
- 34.8.2 Digital-pixel focal-plane arrays
- 34.8.3 Scientific imagers: cryogenic CCDs, sCMOS, and gigapixel mosaics
- 34.8.4 Stacked and processing-in-pixel sensors
- 34.8.5 Curved focal planes
- 34.8.6 Beyond the visible, and filter-array sensors
- 34.8.7 Radiation-hardened and defense focal planes
- 34.8.8 The chapter's point
- 34.9 Extra sensors and non-visual data
- 34.9.1 Accelerometer and gyroscope: the inertial measurement unit
- 34.9.2 Sound: microphones, audio-visual sync, and the visual microphone
- 34.9.3 GPS: geotagging and place
- 34.9.4 Compass and magnetometer: heading and orientation
- 34.9.5 Near-infrared: the cut filter, dark flash, and NIR-assisted denoise
- 34.9.6 Temperature: dark-current compensation
- 34.10 Ultra high-speed imaging
- 34.11 Hyperspectral imaging, color wheels
- 34.12 Polarization imaging