Book Notes Slides P-sets ⌂ Syllabusdraft · v0.2.49.1

Computational Photography

INTRO From digital to computational photography Computation is the new optics The goals of computational photography What makes a technique useful Hand-designed algorithms, learning, and generative AI Does photography still matter in the age of generative AI?A multidisciplinary field Beyond photography How to read this book Choosing how much to read Machine-readable by design How this book was made What programming language for computational photography?On the desktop: Python, C++, and Halide In the browser: JavaScript and the web stack On the phone: Android and iOS Libraries Vibe coding Programmability, or the lack thereof Problem sets FUNDAMENTALS OF IMAGING Light and optics Light: rays, waves, and the spectrum How light is created Light power and brightness Summing coherent vs. incoherent light Reflection, refraction, and what happens at a surface Light-matter interaction is largely resonance Polarization The color of objects: illumination times reflectance Where color comes from The BRDF and the look of materials Radiometry: radiance, irradiance, exposure, and falloff Global illumination Wave effects, diffraction, and the diffraction limit Pinhole image formation and linear perspective Pinhole imaging and the perspective projection Homogeneous coordinates Camera in a general configuration Intrinsics, extrinsics, and what cropping really does What perspective preserves, and what it destroys Wide-angle distortion: spheres bulge and faces stretch at the edges Photography with focal length: framing, magnification, and compression Depth, ray length, and unprojection Lens image formation Single lenses: refraction and Snell's law Thin lens optics Aperture and the f-number Image measurements as integrals over the plenoptic function The plenoptic function The pixel integral Exposure values (EV) and stops Splitting the integral Attempts to visualize the plenoptic function Depth of field The circle of confusion Depth of field versus depth of focus Choosing the maximum circle of confusion The double cone, and the near and far limits Hyperfocal distance The surprising invariance Background blur beyond the focus plane Defocus is not a blur of the image Sensor size scales depth of field Defocus is integration over the aperture dimension of the plenoptic function Motion blur Blur from subject motion Blur from camera shake How shift-invariant is camera-shake blur?Which dominates: translation or rotation?The hand-holding rule of thumb Shake from the shutter and mirror Panning: fighting subject motion by moving the camera Freezing motion with light The space-time view: motion blur is integration over time Sensors: photosites, CCD vs CMOS The photoelectric effect Photon to number: the photosite Beyond the visible spectrum: near-infrared, thermal, and ultraviolet Analog-to-digital conversion CCD versus CMOS Time integration: the exposure Shutters A taste of modern sensor tricks The impact of sensor size How far sensors have come Noise, signal-to-noise ratio and dynamic range Noise sources The algebra of noise Signal-to-noise ratio: it's about ratios Noise variance is mostly affine in pixel brightness A fuller model of noise: Poisson photon noise plus clipping plus Gaussian electronic noise plus a second clipping Noise as a function of ISO Dynamic range Regimes, and how close we are to the limits Making noise variance uniform VISUAL PERCEPTION AND COLOR The eye and image formation Perceptual color and trichromatic vision Cone photoreceptors and perceptual color Metamers Color blindness Opponent process and the multistage model So what are the primary colors?Color as linear algebra Adaptation and constancy Light adaptation Visual perception is largely invariant to multiplications Contrast Lightness constancy Color constancy Spatial and temporal vision Spatial vision Chrominance spatial vision Temporal vision The visual system attends little and makes things up Attention and eye movements The visual system makes things up Animal eyes The optics: many ways to form an image The pupil: aperture shape as a clue to ecology Amphibious vision: focusing in air and in water The sensor: a retina that is not uniform Seeing in the dark: sensitivity and its tricks Acuity: the finest detail, and how to sharpen it Color vision: opsins remixed Polarization: the axis we are blind to Temporal vision: flicker, and motion first Human vision TLDR The eye is a camera with an uneven sensor Three cones reduce a spectrum to three numbers Primaries are a choice, and the axes are skewed Color blindness is a missing or shifted axis After the cones, brightness and two color differences The eye works on ratios, not levels The eye reports surfaces, not light Spatial vision is band-pass, and measured in degrees Color detail is coarser than brightness detail Vision has a finite temporal bandwidth Vision is foveal, serial, and partly invented What the rest of the book does with this COLOR SPACES AND COLOR TECHNOLOGY Linear color reproduction and CIE XYZ The color reproduction perspective Analysis vs synthesis and non-orthogonality Color matching and the most standard color space: CIE XYZ Chromaticity diagram Keeping track of what the numbers mean Linear RGB White balance and white point Von Kries adaptation and white balance How do you know the illuminant?White balance with mixed illuminant White point in color space definitions Spectral sharpening within LMS space The limits of white balance, and CRI (color rendering index)Spectral sharpening with general spectra Subjective white balance White balance datasets Encoding: linear, gamma or log Gamma encoding Gamma the contrast operator (gamma adjustment)System gamma: the end-to-end curve Log encoding PQ, the perceptual quantizer HDR encoding When to work on encoded values Color and encoding in real pipelines What gamma buys, and what it costs RGB color spaces Two different words for impossible Reference: primaries, matrices, and transfer functions Advanced color: opponents, perceptually-uniform, and preferred Opponent color spaces and separating luminance and chrominance Tensions: linearity, gamma, and opponents HSV, HSL, and cylindrical color spaces Perceptually uniform color spaces Recap: a zoo of color spaces Color appearance models Preferred colors: skin, sky, and other memory colors Color names and the prototypes behind them Sensing color: multiplexing strategies Multiplexing in time Multiplexing in space: the color filter array Multiplexing across separate sensors: the beam-splitter Multiplexing per pixel by dispersion: color routers and nano-prisms Multiplexing per pixel by tunable absorbers: quantum dots Multiplexing in depth: stacked photodiodes Hybrid strategies Beyond trichromatic capture: full spectrum and multispectral Spectral sensitivity: matching the eye, catching photons, and white balance Synthesizing color Additive vs subtractive Gamut and gamut mapping Display technologies and multiplexing Display families and their viewing conditions Printing and halftoning Color film Simple display calibration Recap: the full color pipeline, color management and ICC The full color pipeline Color management, ICC, and industry standards Soft proofing Color TLDR Physical color Cones, LMS, and metamerism Light and chromatic adaptation XYZ Chromaticity and gamut Linear RGB, and why there are many White balance and white point Gamma is for encoding (and doesn't hurt processing much)sRGB RAW files Log is for high dynamic range, video, and some processing Opponent (luma-chroma) spaces HSV is for UI Perceptually uniform spaces are for differences The full chain: from physical color to perceptual color to color names and memory Color and encoding in the real world What color space and encoding to do what?Color terms Radiometry: physical light, no observer Photometry: physical light weighted by the eye Colorimetry: color via the standard observer Color appearance: perceptual, observer-dependent Color spaces and encoding Chronic confusions, named One word, several meanings PHOTOGRAPHY The camera as an instrument UI and display Anatomy of a modern full-frame interchangeable-lens camera Anatomy of cell-phone cameras Camera versus phone Types of cameras Cameras beyond photography A camera's other sensors Exposure and capture timing Basic photography: exposure settings (shutter, aperture, and ISO)Exposure modes, UI, and auto-ISO Metering: from printed guides to the sensor f-stops versus T-stops Video capture settings Choosing one frame or several Viewpoint, lenses, and focus Lenses and focal length Aperture and the diaphragm Focusing the lens Choosing and checking autofocus Image stabilization Keeping the lens clean Lens filters: polarizers, ND, and graduated ND Working with light Natural illumination Flash Flash metering Indirect illumination and bounce flash Multiple-point lighting Developing and editing a photograph The traditional darkroom The digital darkroom What a photographer actually does with those sliders Preparing photographs for viewing Why the display characteristics matter Distance, resolution and acuity Sharpening for the display: size and distance Light level viewing conditions Viewing position and perspective Photographic situations and tradeoffs A moving person indoors A backlit portrait An interior with a bright window A close-up object A phone document capture Adapting the cases: wildlife Adapting the cases: night and sky When the intended result is abstract When the apparatus changes the problem One frame, or a body of work Photographic representation and its limits Compensation, accentuation, conflict The picture is flat: depth and its cues The picture is static: time and motion One viewpoint, and a finite frame The contrast is limited: dynamic range and gamut Resolution depends on the task Authorship and what the photograph claims Faithful is not the same as realistic Choosing the instant, choosing the viewpoint Time and focus select information Filters and lighting: building the scene's light The darkroom: tone, dodging, and burning Make-up and retouching: authoring the subject Getting closer: systematic capture and the biases that remain Photography TLDR LINEAR METHODS AND FOURIER ANALYSIS Introduction to forward and inverse linear methods A one-dimensional blur, forward Running the blur backward: the idealized case The catch: noise and ill-conditioning A corner camera: reading the edge of a shadow A different challenge: sampling and aliasing What the measurement determines Convolution Convolution as a weighted sum of neighboring pixels The flip: where-from vs. where-to Convolution as a sum of translated images Properties: the impulse, normalization, symmetry, commutativity Nitty-gritty: finite images A blur zoo Separability Linear sharpening Gradient and oriented filters Is imaging blur usually a shift-invariant convolution?Fixed and data-dependent weights Fourier Why Fourier Complex waves are eigenvectors of shifts and convolution The tiniest Fourier transform in the West Definition: one coefficient per wave Reading an image's Fourier transform A small bestiary of transforms Optics and Fourier MTF: a lens's frequency response The fine print: two limitations of Fourier Windowing: the price of measuring a real spectrum Application preview: can we deblur, given the blur?Convolution by FFT Sampling, downsampling, and aliasing Aliasing: a high frequency in disguise Nyquist and the sampling theorem In two dimensions, aliasing has a direction Sampling in the frequency domain: spectral replicas Seeing it: sampling and pre-filtering, hands on The ideal reconstruction filter: sinc, and why it is unreachable Practical filters: from the box to bicubic and Lanczos Resampling and upsampling Domain operations: moving pixels around Start with scaling up The naive idea, and why it leaves gaps Loop over the output, use the inverse transform Nearest neighbor Linear interpolation, in 1-D first Bilinear, in 2-D The convolution perspective The full signal-processing view: sample, reconstruct, prefilter, resample Upsampling vs. downsampling: scale the kernel Prefiltering when the transform isn't a clean scale Resampling and encoding Beyond Nyquist-limited reconstruction Practical implementation choices Pyramids and wavelets Halfway between space and frequency The idea: process an image at many resolutions The Gaussian pyramid The Laplacian pyramid Reconstruction: an exact encoder and decoder Each level is a frequency band What the bands look like on a real image Limitations Cousins: wavelets and steerable pyramids Pyramid blending Other applications Coarse-to-fine methods Image formation as a linear system Linear systems from a billboard to an image How solvable is an imaging problem?Regularization Linear systems from ray space (plenoptic function or light field) to an image Space–time: imaging across time Other forms of linearity, and multilinearity Designing measurements for recovery When linearity breaks down Efficient linear solvers The problem, and the one trick Direct methods, and why they wall out Basic iterative solvers Conjugate gradient Preconditioning Multigrid Which solver when, and the edges PROBABILISTIC APPROACHES AND MACHINE LEARNING Imaging as Bayesian inference with probabilistic priors Priors and the manifold of natural images Probability: the Bayesian view Optimization, inference, loss Deep learning, amortized inference, and generative AI When you can design the operator, and a teaser of information theory Matching the estimate to the task Harder inverse problems: factorization Machine learning The framing: learned operators replace hand-designed ones The data story: synthetic data, noise models, datasets Deep learning Low-level learned operators (pixel-to-pixel)Mid- and high-level learned predictors Generative models: image-to-image translation Learned perceptual metrics and losses Generative AI and diffusion The framing: generation is learning and sampling a prior $p(x)$Diffusion: generation as iterated denoising Conditioning: text, images, and control Posterior sampling: generative priors for inverse problems Other generative families, in brief Caveats and ethics Image priors A catalog of priors: from smoothness to learned The Gaussian prior in closed form: the Wiener filter Shift and scale invariance Anisotropy: sparsity in a single direction Linear color models Denoising as a universal prior: Plug-and-Play and RED Denoisers and noise-dependent scores Score-based models, the thin manifold, and imperfect priors Information theory The basics, at a high level Compression: the clearest application The imaging system as a channel Information-theoretic quantities as objectives The modern revival: learned compression and generative models Sampling, sparsity, and histogram entropy Where it is least relevant: aesthetics and perception BASIC IMAGE PROCESSING AND ISP Image representation An image is an array Float vs 8-bit vs more bits Stochastic quantization and dithering How the array sits in memory Pixel coordinate conventions Alpha and extra channels Video and more dimensions Pixel lookups and boundaries Three kinds of operation What the numbers mean What the numbers actually mean: encoding What the numbers actually mean: color spaces Tone point operations Three kinds of operation The point-operation curve Exposure: a multiply in linear light Brightness vs. exposure Contrast: steepen about a pivot Black point, white point, and levels The general tone curve Lookup tables Local point operations Beauty curves: the camera "look"Why scene-linear looks dull The shape of the curve Highlights: roll off to white, not to a color Saturation, and the rest of the "look"Vignetting and grain, added on purpose Our curve, in code The discipline Beauty curves in real ISPs Perceptual basis for beauty curves Color point operations Basic color enhancement: saturation and vibrance The color edits, in code Converting to black and white Video color grading Color beautification and preferred colors Accurate is not preferred More vivid, but bounded: the naturalness constraint Three corridors, quantitatively Two primitive moves: translate and contract The oldest shipped example: NTSC flesh-tone correction The rung's documented failure Semantic gating by object detection Histograms The histogram The histogram depends on the encoding space A histogram is a sampled estimate Histogram equalization Histogram matching Constraining the slope: Ward's histogram-based tone mapping Sharpening Linear sharpening Why it's called "unsharp mask"Sharpen in gamma, linear, or log?Non-linear sharpening Dead pixels Where they come from Finding them Fixing them Auto-exposure and auto white balance Auto-exposure: metering White balance and color constancy White balance on film Automatic white balance File formats and compression The big picture: none, lossless, lossy Data versus metadata: EXIF PNG: the format we read from JPEG: compression by perception RAW files: before the cooking HDR formats: more than 8 bits Modern formats Other formats: TIFF and GIF Recap ISP, non-destructive editing:A basic ISP Two stages this part did not teach Real ISPs Pipeline design and tuning The ISP, evolving: traditional → learned → generative Recap 2: non-destructive editing A raw developer in the browser EVALUATION Developing, testing and debugging The workflow: build, run, look at the picture Principles Test on inputs you can verify by hand Finding good real test inputs Per-algorithm debug recipes Crashes and bounds Vibe coding: writing image code with an LLM The expert's eye as evaluation Judge it yourself first You are the expert, in a narrow sense Training your eye The input set decides what you can see Beware the average, look at the worst case Knowing when to stop, and when to ask Image metrics Full-reference vs. no-reference Why not just L2?PSNR SSIM VDP and HDR-VDP Learned metrics The right metric depends on the task Towards meaningful user studies How we got here Should you run a study at all?Three kinds of user research Comparing outputs, and the checklist that makes it mean something When the tool is the contribution Asking before building The rest of the toolbox Reporting, and calling it what it is Doing it ethically Why these studies fail to replicate EDGES MATTER Bilateral filtering Bilateral filter Robustness Cross / joint bilateral Bilateral grid Bilateral upsampling and learning Anisotropic diffusion and robust estimators Non-local means Edge-preserving optimization — colorization The shift: filtering becomes optimization Colorization: the canonical demonstration Writing down the energy Solving it: a sparse linear system we already own One affinity, two faces: the matting Laplacian connection Local Laplacian filters The wish, and why the bilateral leaves a halo What a Laplacian pyramid hands you The mechanism: a remapping recomputed per output pixel One curve, three edits: enhancement, tone mapping, inverse tone mapping Contrast with the bilateral base/detail split The cost, and the fast version Matting Laplacian The matting problem: compositing, and why it is under-determined The color-line assumption: α becomes linear in color Closed-form matting: the matting Laplacian Spectral matting: eigenvectors of the matting Laplacian An alternative matting Laplacian Beyond matting: spatially-varying white balance Guided image filtering The key idea: the output is a local line in the guide Fitting the line, and the one knob Why it is fast: O(N) regardless of window size No gradient reversal Reading the affinity back out Where it is used Locally adaptive regression kernel (LARK)Change the verb: from averaging to fitting Steering the kernel to the local structure The regression flavor of the affinity LARK as a structure descriptor Poisson image editing Edit gradients, not pixels Seamless cloning: the headline application Reconstruction = solving the Poisson equation Poisson vs. pyramid blending Advanced techniques Seam optimization Find the non-edges: least-noticeable cuts Intelligent scissors / live-wire: the cut as a least-cost path Snakes (active contours): the continuous cousin Graph cut: globally optimal boundaries by min-cut / max-flow GrabCut: interactive cutout from a single rectangle Normalized cuts: the spectral relaxation Seam carving: content-aware resize by DP Time-lapse via DP: a seam through time Video textures: looping a seam in time GraphCut textures & photomontage: cut, then blend Where this connects: MRFs and beyond Recap: which edge-aware technique when?The three relationships to edges Pros, cons, and cost: when to reach for each A worked chooser WARPING, MORPHING, AND LAGRANGIAN APPROACHES Warping Warping is a domain transform: move where, not what Forward vs inverse warping: why output-driven wins Resampling: a quick reminder Specifying the warp I: parametric models and the degrees-of-freedom (DOF) ladder Specifying the warp II: free-form warps from sparse correspondences Specifying the warp III: feature lines (Beier–Neely)Liquify: brush-driven warping Antialiasing for complex transforms When the footprint stops being a square: the warp Jacobian EWA: the elliptical weighted average Feline and anisotropic MIP probing: EWA on a GPU budget Other footprint integrators: summed-area tables and ripmaps Footprint estimation, clamping, and when to reach for it Jacobians of common warps Morphing Why a cross-dissolve isn't enough: the ghosting motivation Two interpolations: domain (shape) and range (color)The morphing recipe (combine both)Field morphing: the Beier–Neely line-pair warp Mesh / triangulation morphing (the alternative warp)View morphing: the geometrically-correct in-between of two views Recap and significance Morphable models Step 1: dense correspondence, the precondition for averaging shapes Step 2: Procrustes alignment, factor out pose so PCA sees shape Step 3: PCA, a mean and a basis of eigen-deformations Step 4: shape and appearance are separate bases Step 5: fitting a new photo, analysis-by-synthesis Step 6: once fitted, edit by moving in the space Step 7: from linear PCA to neural priors Recap and significance Shape-preserving warping Why plain interpolation shears: the rigidity gap As-rigid-as-possible (ARAP) manipulation Linear blend skinning and why the weights decide everything Bounded biharmonic weights (BBW)ARAP interpolation: rigid-as-possible morphing between two poses Moving least squares: the simpler cousin Recap and significance Perspective distortion and its correction Keystoning is projection, not a lens flaw The fix is a homography: re-render the façade fronto-parallel The optical alternative at capture: tilt-shift / Scheimpflug Cropping wide-angle photographs: recenter with a homography, don't just crop A different perspective distortion: wide-angle portraits, and a content-aware fix Beyond faces: correcting arbitrary objects (MaDCoW)A montage has no single viewpoint: a family in a box Recomposing perspective after capture: computational zoom The catch: resampling cost, and "only a plane rectifies exactly"Where this sits: one map, then transport Robustness of perspective to the viewer's viewpoint MATCHING PIXELS AND HUMANS ACROSS SPACE AND TIME Brute force Why you must align first Brute-force translational alignment (SSD / NCC)Phase correlation: the whole shift from one FFT Coarse-to-fine alignment on a pyramid Sub-pixel matching Refining the cost: fit a parabola Refining in the frequency domain: phase correlation Refining by gradient: the Lucas–Kanade step What breaks it: peak-locking, texture, and noise One tool, everywhere downstream Sparse matching Where to look: corners, and the structure tensor Invariance: surviving scale and rotation Describing a neighborhood: SIFT and its zoo Matching: nearest neighbor in descriptor space Feature tracking Tracking vs dense flow: sparse-but-long vs dense-but-short KLT = Lucas–Kanade, per feature, iterated over time Good features to track = where the structure tensor is well-conditioned What breaks long-term tracking: drift, appearance change, occlusion, and re-detection Modern point trackers (briefly)An application: synthetic motion blur from a track Robustness: the ratio test and RANSAC The ratio test: reject ambiguous matches before fitting RANSAC: fit from a minimal random sample, score by consensus Variants and degeneracy Deep learning approaches to sparse matching Learned local features: SIFT's job, done by a network Learned matching: reason about the whole set at once The 3-D-aware turn: pointmaps subsume matching Fast matching Approximate nearest neighbors for sparse descriptors PatchMatch: randomized dense correspondence by propagation Fast high-dimensional matching by random projection Optical flow What optical flow is: and whether it is even well-defined Brightness constancy and the optical-flow constraint The aperture problem Lucas–Kanade: local constant-flow least squares (and the structure tensor)Horn–Schunck: global smoothness regularization Large motion: coarse-to-fine warping Learned flow: RAFT (neuralize the classical pipeline)Deep learning approaches to optical flow The unrolling principle: neuralize the classical solver Cost volumes and warping inside the net RAFT and the recurrent update Face tracking Detecting the face Landmarks: pinning down the features Tracking across time Lifting to 3-D: the morphable model Recognition, and the dark side Which library to use Body pose estimation Top-down vs bottom-up On-device, real time Lifting to 3-D: parametric bodies Which library to use DENOISING Noise models, transforms, and benchmarks The noise that is actually there Choosing the working domain Variance-stabilizing transforms A ladder of difficulty Salt and pepper is a different problem Benchmarks, and what a leaderboard number means Denoising by averaging Why averaging works: the $1/\sqrt N$ derivation When the plain mean is wrong: robust combination Calibration frames: what averaging can't fix Handheld low light: the phone in your pocket Deep-sky astrophotography: averaging at the extreme When is splitting the light free?Gaussian, median, and bilateral filtering Gaussian smoothing, and the assumption inside it The median, and the noise it is actually for The bilateral filter: a neighborhood that looks at the pixels Denoise color more than brightness What each filter destroys The noise these filters were shown with, and the noise a sensor makes Pyramid and wavelet denoising The method: shrink the bands of a pyramid Sparse in the right basis Thresholding, hard and soft Shrinkage as a MAP estimate Ringing, and why the transform is usually overcomplete Coring band by band, in practice Coring and the Gaussian scale mixture Total variation and optimization-based denoising Why the penalty's shape decides the answer Isotropic, anisotropic, and the staircase Solving it Where it stands now Non-local means and BM3D A photograph repeats itself Non-local means: similarity as the weight BM3D: group, transform, shrink, aggregate The ceiling of hand-designed priors Learned denoisers DnCNN: learn the noise, not the image FFDNet: hand the network the noise level DRUNet: a denoiser built to be reused Unrolling: from learning a function to learning an algorithm Denoising without clean data CBDNet: estimate the noise, keep the pairs Noise2Noise: two noisy observations Noise2Void and Noise2Self: one noisy image Blind2Unblind: recovering information hidden by the blind spot Choosing supervision for a real camera Generative denoising with diffusion models Why the posterior mean can blur detail Sampling instead of averaging The reverse process is a sequence of denoisings What a hallucinated detail costs Video denoising Temporal redundancy is the resource Patches in space and time Learned video denoisers Temporal consistency, and the metric that does not measure it Denoising as a universal prior Plug-and-play: a denoiser inside ADMM Reusing the denoiser across measurement models RED: the denoiser as an explicit regularizer Tweedie's formula, and what a denoiser has been computing Sampling with a denoiser-derived score DRUNet as a plug-and-play prior, and continuation Measuring a denoiser The reference metrics, and what each rewards Comparing operating points What to hold fixed Ablations, cost, and what to report The question a benchmark does not answer The limits of denoising Denoising TLDR SUPER-RESOLUTION AND UPSCALING What can super-resolution actually do?What the sensor kept and what it folded The forward model, and why this is an inverse problem Two operations, one word What a resolution number means once a prior is involved Multi-frame super-resolution Sub-pixel shifts are extra samples The classical formulations Registration is the hard part What actually ships The gain, and where it stops Learned burst methods Aligning in features, merging by attention A burst is a set, not a sequence Where the training data comes from What a learned merge does that a kernel cannot, and what it costs Single-image super-resolution: classical methods Interpolation, and what it actually assumes Example-based super-resolution Neighbor embedding and the manifold assumption Sparse coding and dictionary learning RAISR: a bank of learned filters, chosen by hashing The image as its own database Single-image super-resolution: deep learning The architectural line Where the upsampling goes The ceiling, and what it was made of What these networks do and do not learn The perception-distortion tradeoff and perceptual super-resolution The tradeoff, stated Buying perceptual quality: perceptual losses and adversaries Measuring on both axes The hallucination problem Generative super-resolution Conditional generation Sampling against averaging Consistency, faithfulness, and what to report The circle closes Reference-based and domain-specific super-resolution Reference-based super-resolution Face and text restoration One model, any scale Learning from the test image alone The ordering, stated Blind and real-world super-resolution The real degradation chain Degradation modeling: the fix was data Estimating the degradation instead of covering it Real paired data, and the gap that remains Video super-resolution and practical systems Temporal consistency is the new constraint Using the neighbors Hybrid space-time super-resolution What is deployed What super-resolution actually delivers Super-resolution TLDR DEMOSAICKING AND JOINT METHODS Basic traditional demosaicking Downsampling and bilinear interpolation, and how they fail Interpolate along the edge, not across it Interpolate differences, not channels Demosaicking and other aspects of imaging Advanced traditional demosaicking Adaptive homogeneity-directed demosaicking (AHD)Residual interpolation and ARI Demosaicking as demultiplexing: frequency-domain methods Where the classical methods stop Mosaics and aliasing Each channel lives on a sparser lattice All channels at once: luma and chroma False color and moiré as spectral overlap X-Trans, Quad-Bayer and Nona-Bayer in the Fourier domain Prefiltering a mosaic: one blur for three lattices Anti-aliasing filters are disappearing Nano-prisms as anti-aliasing filters Optimizing and searching classical demosaickers Fitting a classical demosaicker by gradient descent Searching for the program itself Alternative color filter arrays Fuji's X-Trans Quad-Bayer and its modes RGBW and sparse color Designing the array and the algorithm together Learned demosaicking The data problem hiding inside a free training set Architectures, briefly Unrolling, and the bridge from the classical formulation What the networks learned Benchmarks and their age Neighboring problems in the raw stage Defective pixels Phase-detect autofocus pixels Black level, clipping, and where pink highlights come from Lens shading and crosstalk The ordering claim, once more Joint demosaicking and denoising The two orderings, made visible The joint formulation FlexISP: one energy, one prior What it buys, and where it is deployed Correcting chromatic aberration jointly The argument generalizes Joint methods and the end-to-end pipeline What end-to-end learning gains What it costs Keeping a useful intermediate representation Where the field actually is Where demosaicking sits in the pipeline Quad-Bayer and Nona-Bayer processing Why the classical methods do not transfer Remosaicking: making the sensor look like Bayer Demosaicking the grouped layout directly Remosaic or demosaic directly?Split exposures within a block Synthetic data, unprocessing, and raw reconstruction The inversion, step by step Why an approximate inverse is good enough Raw reconstruction as a goal of its own The pipeline diagram, walked backwards Hardware alternatives to the mosaic Pixel shift: spending time, precisely Foveon and stacked photodiodes: spending silicon depth Assorted pixels: spending spatial resolution on another axis RGB-NIR: adding a band the eye does not have Color wheels: spending time, sequentially, for many bands Four ways to multiplex color Routing color instead of filtering it One table, one question Demosaicking TLDR DEBLURRING What blur is, and what deblurring can undo Why naive inversion fails The fix, stated once Which space to deblur in Non-blind deconvolution The Wiener filter: the regularized, noise-aware inverse Richardson-Lucy: iterate toward the likelihood Constraints instead of priors Priors for deconvolution Sparse gradients: the prior that keeps edges sharp Solving it, and the trick that made it fast Blind deblurring Estimating the kernel and the image at once Can you measure the kernel instead?Why the obvious optimizer fails Two ways to solve it Spatially varying blur Rotational camera shake A table of blurs, and which are convolutions What ships in a phone Defocus, and undoing what the lens softened The defocus PSF, and why one deconvolution is wrong Estimating the blur field Lens softening, which is a different problem with the same shape Engineering the blur instead of fighting it Learned deblurring Networks that skip the kernel The datasets are the argument What learning did and did not change Video, bursts, and lucky imaging Atmospheric seeing: why one long exposure fails Shoot thousands, keep the sharpest Align and stack the survivors A poor man's adaptive optics Measuring a deblur What went wrong, specifically What to measure On what data What to report Deblurring TLDR HIGH DYNAMIC RANGE CAPTURE Dynamic range and multiple-exposure capture The gap between the scene, the sensor and the screen Bracketing, and which parameter to vary Doing it in the sensor instead Veiling glare, the limit bracketing cannot remove Two caveats: single-shot range, and where this is used Camera response and radiance calibration The linear case: recovering the exposure ratios The non-linear case: recovering the curve The pipeline end to end From exposure to photometry Merging exposures The weighted merge When every exposure is bad at a pixel What comes out is not a picture Optimizing the capture and merge Ghosts and motion Align first, and know what alignment cannot do Rejection: pick a reference and disagree with it The other approaches, and why rejection is the usual choice The bracket you did not have to take Single-shot range and inverse tone mapping Range without a bracket, in the sensor Inverse tone mapping: inventing the range Choosing among the three routes HDR+ and burst imaging on phones Why a phone shoots a burst, and why it underexposes The HDR+ pipeline: align and robust-merge in raw From burst HDR to burst super-resolution Encoding HDR images How many bits does a range need?Floating point: half floats and OpenEXR Shared exponents and logarithms: Radiance RGBE and LogLuv Editable HDR: floating-point DNG and TIFF Backward-compatible HDR: from JPEG-HDR to gain maps Display-referred encodings: PQ, HLG and video Choosing an encoding TONE MAPPING AND DISPLAY Tone mapping and perception Why a faithful copy looks flat: four causes The four do not multiply What local tone mapping relies on Global tone mapping Two families: global and local Scale factors and logarithms: Ward and Drago One curve for everything: Reinhard's photographic operator Histogram adjustment: a curve shaped by the image Why log space helps From luminance to color Why a single curve cannot win Choosing and checking a tone map Two-scale and bilateral tone mapping Starting with global tone mapping Base and detail LCIS: diffusion that stops at edges Bilateral tone mapping The decomposition zoo, and what it fixes Exposure fusion Three qualities, one weight Blending without seams Why it survives Local Laplacian filtering and other multiscale methods Multiscale models of adaptation Subband companding: Li, Sharan and Adelson Local Laplacian filters Pyramids in exposure fusion and phones Gradient-domain tone mapping Attenuating large gradients: Fattal, Lischinski and Werman Why it is hard to control A screened version: anchoring the large scales Tone mapping as perceptual optimization Display-adaptive tone mapping iCAM06: the color appearance route Why they did not take over Display rendering transforms and color From operators to view transforms Per-channel curves and the hue problem The responses: AgX, OpenDRT, ACES 2.0 and PBR Neutral Looks and grading Learned tone mapping, in brief High dynamic range in traditional photography Three places to intervene Combination printing: Le Gray and his successors Reducing the range of the scene: lighting Filters at capture: graduated neutral density and its relatives Exposure and development: the negative's curve The Zone System Printing: paper grade, dodging and burning Masking: the darkroom's base and detail What carried over to computation HDR displays and display mapping What a display can actually show HDR displays and dual modulation Display mapping: rendering for the display at hand Gain maps on displays of different headroom Inverse tone mapping Evaluating tone mapping Why the comparisons disagree What can be measured without a reference What needs people What to report HDR TLDR LAYERS, TRANSPARENCY, COMPOSITING, AND MATTING Transparency and compositing Compositing and alpha channels Harmonization and where the blends live Optical effects beyond alpha Segmentation Segmentation: cutting the object out Blue and green screen matting, chroma key Blue/green-screen matting and chroma keying Matting The fundamental matting equation Traditional matting approaches Deep-learning matting Generative matting FIELD OF VIEW EXTENSION AND PANORAMA STITCHING Manual panorama stitching from multiple views The scenario, and the one rule: rotate, don't translate Refresher: pinhole projection is "divide by depth"Why you don't need 3D: depth cancels for a pure rotation Homographies and homogeneous coordinates Solving for $H$ from correspondences Warping and assembling the panorama Another application: document flattening and merging Automatic panorama stitching from multiple views and feature matching Why not brute force, and the two sub-problems The feature pipeline, recalled RANSAC for a homography Blending Why a hard seam is visible: the photometric mismatch Feathering / alpha blending, and why it ghosts Two-scale blending: the simple split (the pset method)Multiband / Laplacian-pyramid blending: a transition per band Poisson / gradient-domain blending: paste gradients, solve for values Seam optimization: route the seam instead of fading it The complete pipeline, end to end Bells and whistles Other projections Bundle adjustment Movement and parallax handling Continuous panoramas (e.g. on cell phones)Incremental registration of a video stream Mosaicking a moving strip (and why a central strip)Rolling shutter and exposure drift INPAINTING Inpainting, texture synthesis, and object removal Inpainting as filling unmeasured pixels: the spectrum of priors PDE / diffusion-based inpainting Texture synthesis (Efros–Leung; Efros–Freeman quilting)Exemplar inpainting: clone, healing brush, and object removal (Criminisi)Data-driven scene completion (Hays & Efros)Deep inpainting (context encoders → partial/gated conv → diffusion)Highlight / specular recovery Epitomes: a compact patch model Patch match The nearest-neighbor field, and why exhaustive search is the bottleneck PatchMatch: randomized correspondence Barnes et al. 2009 Applications: hole filling, retargeting, reshuffle Shift-Map image editing Pritch et al. 2009: editing as graph-cut labeling Inpainting using millions of photographs Why self-similar inpainting isn't enough Scene completion from a huge database The data is the prior, and its modern opposite Generative inpainting Filling a hole against saying what goes in it Inpainting in a latent space, and the seam it creates What it gets wrong Judging a fill with no reference An inpainted region is not evidence COMPOUND LENSES, AND ABERRATION CORRECTION Aberrations and optical challenges Taxonomy of challenges Spherical aberration Coma Astigmatism Field curvature Chromatic aberration Radial distortion Wave effects and diffraction Vignetting Flare and coating Aberrations correction The two families of cure Correction in glass Computational correction Radial distortion correction Aberration data in the DNG, and the limits of per-copy correction Measuring lens quality The modulation transfer function (MTF)Measuring MTF in practice Spot diagrams and the PSF MTF through focus and depth of focus Field-dependent MTF: sagittal vs tangential Color, geometry, and illumination measurements Scalar quality summaries Lens optimization The high-level idea: design as optimization The forward model: ray-tracing and spot diagrams From hand calculation to software Tradeoffs: the design is always a compromise Tolerancing: from the nominal design to a manufacturable one A short bestiary of classic designs The lens as a system: cardinal points, pupils, f-number, T-stop Scaling laws in optics Lohmann's scaling laws: why a good lens is heavy The gigapixel barrier for a single lens The escape: monocentric multiscale optics Capture everything, crop later: the spatial cousin of the light field Special optics Tilt-shift and the Scheimpflug principle Fisheye and non-rectilinear projection Mirrors: catadioptric and reflecting systems Periscope / folded-lens design (smartphone telephoto)Anamorphic optics Stereo (3D) lenses Macro, microscope objectives, and telescopes Teleconverters From shaped glass to thin structures: Fresnel, diffractive, GRIN, metalenses Focus Focus mechanics Focusing a compound lens: beyond unit focusing Focus actuators Focus stacking, macro, and focusing rails Optional background: viewfinder optics Manual-focus aids: making the focal plane visible Optional background: focus control in cinema lenses Autofocus Contrast-detection AF Phase-detection AF (the split-pupil / stereo trick)On-sensor PDAF and dual-pixel AF Depth from focus / defocus, and learned subject AF Where to focus: saliency, faces, and eyes Focusing in astrophotography Glare suppression Where stray light comes from: flare, ghosting, veiling glare Hardware suppression Computational deflare and glare deconvolution Optical stabilization The problem: hand-shake and the blur budget Optical stabilization: lens-shift vs. sensor-shift (IBIS)Digital / electronic stabilization and the computational alternatives The eye as an optical instrument: vision and its correction The eye as a camera Refractive errors: the eye out of focus Measuring the eye's optics Correcting vision Presbyopia and the bifocal problem: from bifocals to AF glasses DEPTH OF FIELD Bokeh, focus stacking, and depth-of-field control The geometry of focus The bokeh look: shape and structure of the blur Extending DoF: focus stacking Controlling and faking DoF Fake (synthetic) depth of field Why phones must fake it Where the depth (or subject) comes from From depth to blur: the thin-lens circle Realistic bokeh: why a Gaussian looks fake Occlusion-aware compositing and matting Failure modes Focal stacks and depth of field extension Why limited depth of field is the problem A simple algorithm: sharpness = local high-frequency energy, then argmax The more advanced method: Interactive Digital Photomontage (graph-cut + Poisson)Capturing the stack: hardware, and the magnification trap ILLUMINATION AND REFLECTION Illumination related effects in a single image Intrinsic images: the unifying frame ($I = R\cdot S$)Multiple-light / spatially-varying white balance Reflection removal — pulling apart a transmission and a glass reflection Shadow detection and shadow removal Specular-highlight removal / "fake polarization" (the dichromatic model)Mixed-lighting white balance Why a global gain must fail Estimating the per-pixel mixture: an under-determined inverse problem Correcting each region for its own light What came after Dehazing Dehazing as a prior-driven inverse problem Differentiable image pipelines and algorithm optimization (Halide)Intrinsic images with time lapse The split, and why one image can't do it Weiss 2001: the median of log-gradients Passive and controlled illumination Colorization One channel in, three channels out A spectrum of priors: scribbles, references, and learned models The multimodality trap: why naïve colorization goes muddy Closing the loop: learned priors with a human's hints Plausible is not correct Where it sits NPR AND STYLIZATION Style transfer Classical style transfer: patches and statistics Neural style and feed-forward stylization Style transfer as image-to-image translation Non-photorealistic rendering What NPR is for, and the one idea Stroke-based / painterly rendering (and the brush p-set)Edge-preserving abstraction: bilateral + Difference-of-Gaussians Example-based stylization and the bridge to neural style Region-based stylization: stained glass, low-poly, mosaics Artistic screening and halftoning AI FOR PHOTOGRAPHY Editing as a program The edit as a recipe Learning a sequence of operations, without language Predicting a curve against predicting a plan When pixels are the right output Language as an editing interface What people say, and what tools do From a phrase to an operation and an amount Regions: "the sky", "her face"Asking back Language models that call vision tools A language model as a dispatcher Writing a program over image modules What these systems cannot see Planning a retouch Teaching a model to see an adjustment A plan with reasons An agent over a real editor's tools Editable output Verify and correct Decompose, act, check Who does the checking Stopping AI editing in products Driving the tools already there Describing the result instead Assistants that call an editor Trust, authorship and evaluation Did it do what was asked?Can the user see what it did?Does the photograph still say what it recorded?GENERATIVE IMAGE EDITING PHOTO COLLECTIONS Photo mosaics The tiling-and-matching pipeline Color correction and avoiding repeats Multi-scale and irregular tilings Why it resolves into the target at a distance Retrieval Classic CBIR: histograms and the text-retrieval analogy Deep retrieval: learned embeddings and CLIP Retrieval at scale: approximate nearest neighbor Mining what makes a place distinctive Auto curation Technical quality: the easy rejects Aesthetics: the hard, learned part Grouping, summary, and diversity Arranging the collection after culling Life logging cameras The devices and how they fire The memory-prosthesis reframing The big-data problem passive capture creates Privacy and ethics Photo tourism Structure-from-motion on internet collections From reconstruction to experience Lineage Photobios Align and order the collection Let the data fill the gaps Collection as experience Average Explorer Galton's composite portraiture: what it was for Alignment is everything The average of a category An artistic lineage AverageExplorer: averaging made interactive What an average is, and what it is good for Pix 2 GPS Geolocation as retrieval over a geotagged corpus The answer is a distribution Mapping the collection itself The learned successors Personalized priors Personalized restoration Personalizing generative models The bargain and its ethics Artistic projects with photo collections Statistical collage: Salavon Anticliché camera Pareidolia Displaying images together Selection Layout Color and coherence Themes MAKING COMPUTATIONAL PHOTOGRAPHY CODE FAST Algorithmic speedups Precompute and look up: pay once, before the loop Separability: a 2-D pass for the price of two 1-D passes Recursive / IIR filters: a running state, cost independent of radius Integral images / summed-area tables: any box sum in four lookups Fast median filters: a sliding histogram for a constant-time median Pyramids and multiscale: do the large-scale work on small images Discretize the range to accelerate non-linear filters Downsampling and edge-aware upsampling Choosing a method Modern CPUs: memory hierarchy, parallelism, and what it takes to go fast Why moving data, not doing math, is the bottleneck The memory hierarchy and locality The roofline: is my kernel compute- or memory-bound?The forms of parallelism Measuring and improving a kernel Why photography is hard for the machine Keep the comparison valid Halide at a high level: decoupling algorithms from schedules Images as pure functions The schedule: four questions for every stage What the compiler derives What the split buys, measured How long does a schedule take to write?Where it ended up Letting the compiler find the schedule The split was always there, done by hand Gradient Halide: differentiating the pipeline When to use the split Halide programming Getting started with Halide Functional, with no loops: Func, Var, Expr A first image pipeline: brighten, then blur Reductions: RDom, sums, and histograms The two extremes: root and inline Scheduling the loops within a stage: reorder, split, tile, vectorize, unroll, parallel, fuse The heart of it: producer-consumer granularity (compute_at, store_at)Comparing schedules with measurements Boundaries and bounds Same algorithm, new machine: the GPU Letting the compiler schedule: the auto-scheduler A repeatable scheduling experiment Scaling up Halide Decoupling storage from computation: store_at A 7×7 box blur: reductions and update definitions Harris corner detection: ten stages, fifteen edges A brute-force bilateral filter: what a schedule cannot fix The bilateral grid: changing the algorithm instead A three-level Laplacian pyramid: multi-resolution stages The development loop: correctness, measurement, and benchmarking hygiene Going farther: what the tutorials cover that we did not Hardware backends: GPU, NPU, DSP GPU: the data-parallel workhorse The neural accelerator: NPU and TPU DSP: the real-time control loop ISP, FPGA, and ASIC: the fixed-function end The heterogeneous SoC, and the spectrum to carry away On-device ML runtimes, and why the work stays on the phone Efficient neural network inference Quantization: fewer bits per weight Pruning: fewer weights Knowledge distillation: a small student, a big teacher Low-rank and tensor factorization Efficient architectures: cheapness designed in Neural architecture search: let the machine design it Hardware-aware deployment and the roofline Comparing deployable models Automatic search for fast methods The problem is a curve, not a point What a broader search space can offer Ma et al. 2022: searching structure and parameters together Three axes of "search instead of design"The broader family, and what it costs From search result to implementation 8-bit and fixed-point arithmetic Integer versus fixed-point: where the binary point sits Rounding, dithering, and the banding trap Saturation and the width of the accumulator fp16 versus bf16: the exponent–mantissa bargain int8 quantization for neural inference Where wider arithmetic matters VIDEO Motion blur, temporal sampling, and resampling A frame is an integral over time → motion blur Time is sampled → temporal aliasing, the wagon-wheel effect Motion blur is the temporal prefilter: the two are one tradeoff The film movie projector Resampling in time: frame-rate conversion Lagrangian vs Eulerian: the organizing distinction for the part Video compression and motion compensation Why video compresses far better than still × N: temporal redundancy Motion-compensated prediction: the core trick I, P, and B frames; GOP structure Why this is "optical flow on a budget"Modern codecs in one breath Video editing Non-linear editing: the timeline metaphor Summarization: keyframes, fast-forward, and highlights Fun temporal filters: reduce-over-time Transcript-based editing In-betweening: synthesizing the frames an edit needs Coda: storyboards, interviews, and where this part lands Frame interpolation and slow-motion synthesis Why interpolate: faking slow-motion and up-converting frame rate Interpolation = morphing between adjacent frames Flow-based interpolation: warp both frames to the midpoint and blend Learned synthesis: Super SloMo and FILM Hybrid low/high resolution and frame-rate imaging The space–time bandwidth trade-off The hybrid two-camera architecture Motion from the fast stream Applications: deblur, space–time super-resolution, video from stills Modern descendants Video stabilization and rolling-shutter correction What stabilization is: a camera-path signal to be smoothed Stage 1: estimating the camera trajectory Stage 2: smoothing the path (low-pass vs. L1-optimal cinematic paths)Stage 3: re-rendering and the stabilization↔crop tradeoff Rolling-shutter correction: per-row pose and rectification Time-lapse photography Capturing a time-lapse: interval, shutter, and day-to-night transitions Deflickering: stabilizing exposure, white balance, and light Hyperlapse: stabilizing a time-lapse that also moves Mining time-lapses from the internet Factoring a time-lapse: reflectance, illumination, and intrinsic images Video textures Finding good transitions Playing it Relatives and descendants LIGHT FIELDS AND PLENOPTIC CAMERAS Light fields 101 Capture rays, not pixels The plenoptic function, reduced to four dimensions Ray and point are dual Reading the 4-D structure through its 2-D slices Rendering a new view by looking up rays Light fields vs. plenoptic and radiance Light field cameras Integral photography: Lippmann's fly's-eye plate (1908)The plenoptic camera: a microlens array on the sensor Lytro: the consumer plenoptic camera Commercial light-field cameras beyond Lytro Camera arrays: a grid of full cameras The spatial↔angular tradeoff Two strategies, and why arrays and microlenses are dual Other light field acquisition setups One camera on a gantry: sample the aperture in time Handheld, unstructured capture: let the poses be irregular Catadioptric capture: one sensor, many viewpoints at once Coded aperture in time: sweep the pupil itself The everything-else, mapped Refocusing and synthetic aperture Reconstructing a photo: it is all about which rays you sum Digital refocusing is shift-and-add A focal stack from one capture, and an all-in-focus image Fourier-slice photography: the fast version Reading refocus and depth off an epipolar slice Synthetic aperture: an aperture the size of a room Aberration correction in light fields An aberration is misrouted rays Re-routing each ray to the ideal-lens position The trade: ray bookkeeping instead of glass, and only what you sampled Light field aliasing and 4D Fourier analysis The light field is a sampled signal, and its samples are viewpoints Depth is slope is spectral orientation The bowtie: a spectrum shaped by the scene's depth range How densely must you sample? The plenoptic-sampling bound Geometry buys back samples: the depth-vs-views tradeoff Under-sampling looks like a ghost Where this sits Lumigraph and shape priors for sharper light field rendering Pure light-field rendering blurs because it has no shape The Lumigraph: reproject onto a geometry proxy, then blend Unstructured inputs: free-hand views, no grid required Surface light fields Light field microscopy The optical setup: a microlens array at the intermediate image plane The spatial-versus-angular trade: a coarse 3-D volume Recovering the volume: synthetic refocusing, then 3-D deconvolution Why single-shot 3-D is the whole point Light field networks NeRF: a radiance field rendered by volume integration Light field networks: a ray straight to color, in one evaluation A family of neural light fields Test-time training: the network as the per-scene prior Hand-off: from rays to radiance fields and generation Practical aspects of light field cameras Do you lose all that resolution?Can you do video? Is it practical? The bandwidth problem The time dimension is high speed Adjacent frontiers, briefly So, do I get my camera?MULTI-APERTURE IMAGING Camera arrays: one rig, many instruments One rig, four instruments Commercial arrays: the Light L16 Where this sits Bullet time Multi-camera phones COMPUTATIONAL OPTICS AND CODED IMAGING Wavefront coding Why you cannot just deblur defocus The fix: re-engineer the blur The cubic phase plate The depth-invariant PSF and a single deconvolution What it costs Where it sits Compressive sensing Sub-Nyquist: fewer measurements than unknowns Two ingredients: incoherent measurements and sparsity Recovery by $\ell_1$: the geometry of basis pursuit Why it works: the restricted isometry property, intuitively The single-pixel camera Where compressive sensing wins, and where it does not Coded aperture The bad forward operator of a clear aperture A mask designed for a flat, zero-free spectrum One shot, two outputs: depth and an all-in-focus image The design criterion: which pattern?Coded-aperture pairs: splitting the trade across two shots Heterodyning the light field: dappled photography Where it sits Phase-coded apertures A reminder on wavefront coding Focus sweep The lattice-focal lens Where it sits Code in time (phase, amplitude)Why ordinary motion blur is (almost) unrecoverable The flutter shutter: chop the exposure into a code The decode: one deconvolution, a sharp moving object Amplitude in time, and phase in time Coded strobing, temporal multiplexing, and compressive video Motion-invariant photography Where it sits Theoretical analysis of imaging systems in the 4D light field Fourier domain The light field's spectrum, and the one move that explains everything Every camera is a different slice Putting cameras on one footing: the Bayesian comparison The dimensionality gap: most of the 4-D spectrum is afocal The lattice-focal lens: tiling the wedge The upper bound, and the gap we have not closed From cameras to light transport: the same spectral lens Where this leaves the part End-to-end optimization The pipeline as one differentiable graph Backpropagating into the glass What gets designed: a height map, not a hyperparameter A gallery of deep-optics results What it costs, and where it can go wrong Where it sits Fourier optics Light as a wave: amplitude, phase, and what "coherent" means Diffraction is a Fourier transform A lens computes a Fourier transform The pupil function is the transfer function The diffraction limit Aberrations are pupil phase Every code in this part is a choice of pupil Fourier ptychography: synthesizing a bigger pupil Exotic / advanced optics Lensless imaging GRIN Metalenses and advanced crazy optics à la Barbastathis (coherent though)Non-linear optics Optical modulators (spatial light modulators): DMD and LCD/LCoS COMPUTATIONAL SENSORS Assorted pixels Dual-pixel and phase-detect pixels: buying depth and focus Clear, white, and other color-filter variants Polarization pixels Spatially varying exposure: assorting for dynamic range The common thread Modern sensors (quad Bayer, in-sensor HDR, and beyond)Quad Bayer, Tetracell, and nona-binning In-sensor HDR: capturing range before the merge Dual-pixel autofocus, on the same sensor BSI and stacked sensors: compute under the pixels The global-versus-rolling shutter trade Beyond: nano-prism, organic, and event pixels On-sensor HDR Staggered / multiple-exposure readout (DOL-HDR)Dual (and triple) conversion gain (DCG)Split-pixel: a large and a small photodiode Spatially-varying exposure (SVE) / assorted exposures Lateral overflow integration capacitor (LOFIC)Logarithmic, self-resetting, and counting pixels On-sensor HDR versus multi-frame HDR Depth sensors Stereo: passive triangulation Structured light: projecting the texture LiDAR and direct time-of-flight Time of flight Passive depth from one camera, in passing The menu, in one view Single-photon sensors (SPAD, avalanche, photon counting)From avalanche gain to a single-photon click Photon-counting arrays: zero read noise, shot-noise-limited The Quanta Image Sensor: a different road to one photon What it costs: dark counts, dead time, fill factor, data rate Doppler / velocity imaging The Doppler shift, as a velocity sensor The instruments: vibrometry, Doppler LiDAR, radar Where it fits Event sensors How it works: per-pixel change detection The upside: microseconds, dynamic range, no blur, little data The downside: no picture, and a new kind of data Uses, lineage, and the contrast with single-photon Specialized and research sensors Geiger-mode avalanche-photodiode arrays: photon-counting laser radar Digital-pixel focal-plane arrays Scientific imagers: cryogenic CCDs, sCMOS, and gigapixel mosaics Stacked and processing-in-pixel sensors Curved focal planes Beyond the visible, and filter-array sensors Radiation-hardened and defense focal planes The chapter's point Extra sensors and non-visual data Accelerometer and gyroscope: the inertial measurement unit Sound: microphones, audio-visual sync, and the visual microphone GPS: geotagging and place Compass and magnetometer: heading and orientation Near-infrared: the cut filter, dark flash, and NIR-assisted denoise Temperature: dark-current compensation Ultra high-speed imaging Streak cameras: sweeping time onto a spatial axis Femto-photography: a movie of light in flight Compressive ultrafast photography: a single-shot coded streak Transient imaging and looking around corners The bridge to direct time-of-flight and LiDAR Hyperspectral imaging, color wheels Why three numbers aren't enough: RGB as a 3-sample projection Building the spectral stack: filter wheels, tunable filters, pushbroom, snapshot What it's for: material ID, agriculture, art and beyond Polarization imaging What a camera throws away: polarization as a third axis of light Capturing the stack: rotate a polarizer, or a polarization mosaic Combining images at different polarizations COMPUTATIONAL ILLUMINATION Flash photography Flash / no flash Ramesh's multiflash Removing flash artifacts Dark flash and covert infrared photography High-speed and stroboscopic photography Freezing motion: the microsecond strobe The trigger problem Stroboscopic multiplicity: a sequence on one frame Digital descendants: LED strobes and high-speed cameras Illumination-based matting The well-posed case to beat: chroma key The magic prism: Disney's sodium-vapor process Near-infrared and time-multiplexed matting Flash/no-flash matting: separation by falloff Engineering the capture instead of the prior Separation of direct and global illumination The frequency insight Nayar's program: programmable, structured illumination Toward coherent separation Light domes The reflectance field and one-light-at-a-time capture Relighting by linear combination Scaling down: tabletop LED domes Scaling out: the dome taken into the wild The found dome: the eye Automatic aesthetic lighting Computational bounce flash Drone lighting: flying the light into place Dual photography Light transport as a matrix Helmholtz reciprocity: transpose the matrix The unsettling reach: privacy and seeing the unseen Coherent imaging The confocal principle: rejecting out-of-focus light Seeing through scattering media Illumination as a designed variable 3D AND DEPTH Multiple view geometry Two views: stereo and disparity Two-view geometry: epipolar lines, and the essential and fundamental matrices What "depth" means, and where it comes from Depth means the camera-frame z-coordinate Where depth comes from: a cue-and-sensor inventory Relative vs metric: the scale you usually don't have Monocular depth estimation (one image → depth)Why one image cannot determine depth Relative depth, and the scale-and-shift ambiguity From hand-built priors to borrowed diffusion priors What the maps are good for Single-image 3-D: tour into the picture, photo pop-up, 3-D Ken Burns The universal recipe, and why holes are the hard part Tour Into the Picture: the spidery mesh Automatic Photo Pop-up 3-D Ken Burns and 3-D photos: the monocular form Multi-view 3-D reconstruction: the classic pipeline The pipeline, stage by stage Why it is brittle: and why SfM survives anyway Structured light scanning Projector as inverse camera: triangulation with trivial correspondence The coding ladder, and the frames-versus-motion trade Calibration and failure modes Photos → radiance fields and Gaussian splatting (NeRF, 3DGS)Two goals, one diagram Inverse differentiable rendering, and the discontinuity that forced fuzziness NeRF: a scene as a tiny neural network Do we even need the network?3-D Gaussian splatting: the lessons without the deep learning The practical recipe: and what is baked in Relaxing the assumptions: NeRF in the wild Feed-forward (amortized) 3-D: skip the per-scene optimization Amortization: pay once, reuse forever The line: DUSt3R, MASt3R, VGGT The observation, and the loop it closes The trade, and the data dependency Re-photography Why you cannot just overlay The real-time guidance loop Where it sits The landscape, and is 3-D a "fake task"?The field spans several design axes Is 3-D a "fake task"?The counterpoint, and the frontier INTEGRAL AND IMMERSIVE IMAGING Stereo glasses Wheatstone's stereoscope (1838)Routing a different image to each eye Shooting and synthesizing a stereo pair VR goggles The three levels The optics: a microdisplay and a magnifier per eye Tracking, latency, and why headsets used to make people sick Passthrough, mixed reality, and where today's products sit 3D displays with accommodation Four ways to deliver a focus cue The other gaps, and the ultimate display Lenticular displays Lippmann's integral photography: the common ancestor Multi-view, and the resolution–views tradeoff Light-field telepresence: Google Starline Display depth of field and antialiasing Holography Lippmann and Gabor: recording the wave Off-axis holography and the space-bandwidth wall Computational holography Retinal projection The Maxwellian view: focus set by the display, not the eye From the virtual retinal display to laser eyewear The frontier: writing to individual cones REVEALING THE INVISIBLE Accidental cameras The accidental pinhole: a window is a camera The accidental pinspeck: the anti-pinhole The occluder as a crude lens, and recovery as deconvolution Corners and doorways: an edge that resolves the hidden room Where else the world hides a camera Reflections in the eye The cornea as a catadioptric mirror The geometry: from a corneal pixel to a direction in the world What the recovered reflection is good for Eyes for relighting Motion and video magnification The Lagrangian precursor: track, then exaggerate Eulerian video magnification: amplify the time series at each pixel Why amplifying brightness amplifies motion Phase-based magnification: move the motion into phase What it reveals: vital signs, structures, materials, modes Visual microphone From sub-pixel motion to a sound waveform Bandwidth: high-speed cameras and the rolling-shutter trick How good is the copy? The object's frequency response The active cousins: laser vibrometry and interferometry Corner camera The edge as a one-dimensional aperture From a faint gradient to a usable signal What the corner can and cannot tell you Active non-line-of-sight Third-bounce geometry and time-of-flight The hardware: photographing light in flight From back-projection to fast, exact inversion The trade, stated plainly Passive non-line-of-sight The occluder is what makes it solvable A deconvolution where the lens is unknown One dimension: the corner camera Two dimensions from a single photo: computational periscopy Active versus passive, the ledger Mm-wave, wifi Why radio walks through walls Time of flight, again: radar is NLOS with a longer wave From a radio smear to a human skeleton: the learned map What it sees, and what it costs us ADJACENT FIELDS AND APPLICATIONS Optical computing Astro Extreme long exposure Tracking, stacking, and selection (pointers)X-ray Microscopy Mm-wave Music, sound Fluorescence Opto-acoustic Ultrasound Aerial imaging Computer vision Robotics, driving HUMAN FACTORS Human factors and the art of photography Make better photos Typical shooting scenarios Macro photography Special effect photography Fun artsy stuff Perception of art Ethics of computational photography Computational models of perception Spatial (and spatio-temporal) vision User studies Accessibility: photography by and for blind users Blind camera — capture without a sighted operator The social and personal practice of photography Purposes and subsequent uses A small map of motives Photography as communication and memory What the psychology actually shows When people take photos, and why the usual numbers are not evidence From observed practice to design requirements Optional background: paid photographic work IMAGE FORENSICS AND AUTHENTICATION Image Forensics The problem and the threat model Sensor and pipeline traces: PRNU, CFA, and noise Compression, geometry, and metadata forensics Deepfakes, GAN/diffusion fingerprints, and learned detection Why forensics is evidence, not proof — and the hand-off to provenance Authentication and provenance (C2PA)From detection to attestation: trustworthy cameras and watermarking C2PA and Content Credentials: the standard AI disclosure, watermarking, and the regulatory push Limits, critiques, and the forensics partnership SYSTEMS Programmable and modular cameras What a closed camera hides The Frankencamera: a camera as a pipeline of requests From FCam to camera2 Opening cameras that were not meant to be opened Modular cameras Image processing libraries What a library provides Eager whole-image libraries Demand-driven, streaming libraries Separating the algorithm from its schedule GPU and tensor libraries Choosing Lightroom-style raw developers The recipe and the catalog The pipeline and its order Case study: CompPhotoRoom What a raw developer knows about cameras Photoshop-style editors The layer stack Tiles Case study: gamut Scheduling: pull, cache, fuse What it buys Editors in the browser Networking and image transport Delivering the right image Tiles and pyramids over the network Where to compute Sending a recipe instead of the image Photography programming on phones Android: camera2 and CameraX iOS: AVFoundation, Core Image, and the photo library The closed middle Running the computation on the device CONCLUSIONS, DISCUSSION Recap in context Modern phones, multiple apertures, pano, HDR+Recap: a modern mirrorless camera Recap: a modern cell phone multi camera Lightroom Photoshop Why phones are so good (at photography)Computation beats glass The whole pipeline is co-designed Machine learning and data at scale Many small cameras for one big one The human and system advantages The hard caveats: physics still wins where it must The throughline BACK MATTER Bibliography Glossary Acronyms Term index APPENDICES C++ for Python programmers What changes, coming from Python A first taste, side by side A few more comparisons Files and compilation Value, reference, and const Memory, briefly Images in C++Debugging, and the parts that hurt Where to go next Refreshers Linear algebra Calculus: derivatives, gradients, integrals Optimization and regression Probability and information theory Machine learning and deep learning Programming: Python, C++, and PyTorch Problem Set 0 — Environment and C++ basics Summary Installation and Environment Setup C++Submission Problem Set 1 — Image class, point operations, and color Summary The Image Class Brightness and Contrast More Image Class Methods Colorspaces Spanish Castle Illusion White Balance Submission Problem Set 2 — Convolution and the bilateral filter Summary Smart Accessor Blurring Denoising using Bilateral Filtering Extra credit Submission Problem Set 3 — Denoising and demosaicking Summary Denoising from a sequence of images Demosaicing Edge-based green Red and blue based on green 6.8370 only (or 5% Extra Credit): Sergey Prokudin-Gorsky Extra credit (maximum of 10%)Submission Problem Set 4 — High dynamic range Summary HDR merging Tone mapping Extra credit (10% max)Submission Problem Set 5 — Resampling, warping, and morphing Summary Resampling Warping and morphing Extra credit Submission Problem Set 6 — Homographies and manual panoramas Summary Class morph Homogeneous coordinates Linear algebra Warp and image with a homography Compute homography from 4 pairs of points Bounding boxes Extra credit (up to 10% total)Problem Set 7 — Automatic panoramas Summary Previous problem set code Class morph Harris corner detection Descriptor and correspondences RANSAC Automatic panorama stitching Blending Mini planet 6.8370: Stitch N Images (6.8371: Extra Credit 5%)Make your own panorama Extra credits (10% max)Problem Set 8 — Non-photorealistic rendering Summary Paintbrush splatting Painterly rendering Oriented painterly rendering Your image Paper review (6.865 only)Extra credits Problem Set 9 — Make-your-own, video, and ethics Summary Make your own assignment Ethical issues in computational photography Assignment lists EXIF and image metadata What EXIF is The fields, grouped by what they describe How far to trust it Privacy: the metadata that follows the picture Reading and writing EXIF DNG: the Digital Negative What DNG is, and the problem it solves Inside the container What the raw payload looks like: mosaic vs linear The color recipe: matrices, profiles, and white balance Opcodes: corrections the decoder must apply Compression, and embedding the original Where you meet DNG: adoption and relatives Trade-offs, and DNG's relation to EXIF Rendering a raw DNG The pipeline, step by step Where Lightroom's "look" actually comes from What we would ask an Adobe engineer to check Two kinds of DNG, and the special cases Easy mistakes (most of which we made)Datasets Classification and features Super-resolution Deblurring and restoration Denoising HDR and tone mapping Retouching and enhancement Depth and motion Light fields Color and white balance Faces Inpainting, segmentation, and matting Image quality A camera-feature wish list Exposure, ISO, and dynamic range Bracketing more than exposure Focus and depth of field Computational raw and the sensor Motion data, metadata, and workflow Panorama and multi-shot The interface and the ecosystem How this book was created (the early process)Two documents, not one Compiling a section Figures as code Generative imagery: cover art and 3D Verification and review Keeping a long book coherent The toolchain What the machine did, and what it did not Who wrote what: a per-part estimate How this book is made now From compiling a book to editing one How the author and an agent work together Eleven agents and an ownership map The files that hold it together Skills: writing down what you repeat The gates What goes wrong now What it cost The bundle The course tutor: a local, book-grounded AI teaching assistant What it is, and what it is for Local-first Grounded in the book: retrieval-augmented generation How the retrieval works Indexing other people's books Measuring it, and keeping it from going stale It links, it shows equations, it shows figures Two front-ends, one core What the instructor sees Privacy and disclosure The semi-automatic grading system The shape (to be confirmed)Automatic versus human (to be confirmed)To be filled in (from the instructor)Under the hood: prompts, patterns, and verifiers Prompt patterns that made it work The verifier suite Why this is the interesting part Reading a Lytro light field The container: the LFP/LFR format The pipeline, step by step From hexagonal lenslets to a uniform grid Why a white image Open questions and where we approximate Easy mistakes (most of which we made)File conversion tools JPG → PNG, without an alpha channel DNG → a linear PNG A Lytro capture → a Stanford-style light-field archive How they run in the browser Easy mistakes The interactive figures — a prompt-by-prompt making-of Summary Lorentz resonance: fig-lorentz-resonance (Figure 2.1.8)Rainbow droplet: fig-rainbow-droplet (Figure 2.1.14)Diffraction wave simulation: fig-diffraction-wave-sim (Figure 2.1.29)Exposure-triangle simulator: fig-exposure-triangle-sim (Figure 2.11.5)Repeated quantization: fig-repeated-quantization (Figure 3.1.4)JPEG generation loss: fig-jpeg-generation-loss (Figure 3.17.9)Exposure round-trip (JPEG): fig-exposure-jpeg-roundtrip (Figure 3.17.10)1-D sampling pipeline: fig-sampling-1d-demo (Figure 3.10.4)2-D sampling pipeline: fig-sampling-2d-pipeline (Figure 3.10.6)Mitchell–Netravali bicubic (B,C): fig-bicubic-bc (Figure 3.11.13)Rotation resample challenge: fig-rotation-resample-interactive (Figure 3.11.18)Mini-Lightroom: fig-mini-lightroom (Figure 3.18.4)Poisson blending: fig-poisson-blend (Figure 5.1.5)Bilateral grid (3-D): fig-bilateral-grid-3d (Figure 5.2.15)Perspective montage: fig-perspective-montage (Figure 6.6.13)Beier–Neely morph: fig-beier-neely-demo (Figure 6.3.4)Live face landmarks: fig-face-landmarks-live (Figure 7.10.2)Lens optimizer: fig-lens-optimizer-demo (Figure 9.4.4)Full auto-panorama: fig-pano-stitch (Figure 10.7.9)CLIP-IQA curation: fig-clip-iqa (Figure 11.3.2)Photobio time-lapse: fig-photobio-demo (Figure 11.7.2)Refocus shift geometry: fig-refocus-shift-geometry (Figure 13.4.2)Chroma key: fig-chroma-key (Figure 8.11.7)Portrait-lighting simulator: fig-portrait-lighting-sim (Figure 2.11.26)Aberration explorer: fig-aberration-explorer (Figure 9.1.6)Interactive demos index and search Find a demo Sharing, linking, and embedding a demo Introduction Fundamentals — light, optics, sensors Visual perception and color Color spaces and color technology Photography Linear methods and Fourier analysis Probabilistic approaches and machine learning Basic image processing and the ISP Making computational-photography code fast Edges matter Warping and morphing Matching pixels across space and time Single-image computational photography Compound lenses and aberration correction Multiple-exposure imaging Many images and photo collections Video Light fields and plenoptic cameras Computational illumination 3-D and depth Revealing the invisible Appendices and end matter Not yet placed Course slides The lectures Course slides, 2026 edition Course slides, earlier years 2006, the first offering The ENS / INRIA edition (around 2009)2011–2012, the Keynote editions The MIT year-by-year editions (2007–2015)The Computational Photographer Faculty jobs: why and how to get them Can academia compete with the resources of industry?Family in a box Do good research, run a great lab AI image synthesis is the biggest innovation in image making since photography Lidar for autofocus TedFest 2022 I want a light field camera Lloyd Chambers's feature Christmas list Ethics and computational photography The DSLR will probably die. Are mirrorless the future of large standalone cameras?Eurographics keynote slides Halide talk Do good research Slides from my Siggraph talk about computational photography No telephoto on cell phones A perfect flat lens with metamaterials Why video compression fails on confettis This camera shows you a similar picture instead of the one you are taking Project idea: automatic composite portrait Lytro Cinema Camera: guestimates of characteristics Auto white balance for flash and LEDs Autofocus noise and microphone array How to do research Frédo's own notes A few more starting points Further reading, talks, and advice Odds and ends The Art and Science of Depiction Introduction There is no passive recording What is a picture?Introduction to human vision Vision solves problems Picture organization and Gestalt Gaze movements and focal points Color Limitations and compensations I: the picture is flat Limitations and compensations II: viewpoint and frame The image is static; contrast and gamut are limited Contrast Representation systems Perspective Drawing and line drawing Denotation systems Tone system Non-photorealistic rendering, and style Photoshop 1.0, running What is on the menu bar Running it in a browser What to try The source code is public, and that is not the same as buildable How the program is put together What the assembly relies on Photoshop 1.0 ported to JavaScript Gaussian Blur, and five boxes Unsharp Mask is thirty lines Find Edges, without a square root Add Noise, and the same theorem again Median, and what happens at the border Despeckle and Sharpen Edges are not kernels Custom is the general case, and the built-ins are the special cases On reading source code MISSING STUFF, BUGS fig-learning-transition (hand-designed → learned → generative spectrum) — parked here Denoising by averaging is after basic denoising Photomosaics, my self organizing maps Rephotography Extreme low light Tilt shift,Misc.De-weathering (fog, rain)Near-infrared (NIR) photography (relocated from REVEALING THE INVISIBLE, 2026-06-28)

BACK MATTER

Big lessons Glossary Acronyms Index

show tags

💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.

▲ UpBook contents ⌂ HomeHome Next ▶From digital to computational photography

1.0 Intro⧉

Contents of this part

1.1 From digital to computational photography
- 1.1.1 Computation is the new optics
- 1.1.2 The goals of computational photography
- 1.1.3 What makes a technique useful
- 1.1.4 Hand-designed algorithms, learning, and generative AI
- 1.1.5 Does photography still matter in the age of generative AI?
- 1.1.6 A multidisciplinary field
- 1.1.7 Beyond photography
1.2 How to read this book
- 1.2.1 Choosing how much to read
- 1.2.2 Machine-readable by design
1.3 How this book was made
1.4 What programming language for computational photography?
- 1.4.1 On the desktop: Python, C++, and Halide
- 1.4.2 In the browser: JavaScript and the web stack
- 1.4.3 On the phone: Android and iOS
- 1.4.4 Libraries
- 1.4.5 Vibe coding
- 1.4.6 Programmability, or the lack thereof
1.5 Problem sets

▸ full collapsible outline of this part

▲ UpBook contents ⌂ HomeHome Next ▶From digital to computational photography