29.0 MAKING COMPUTATIONAL PHOTOGRAPHY CODE FAST⧉
A useful camera pipeline must finish within a time, memory, and energy budget. A viewfinder needs sustained frame throughput and bounded latency; an offline editor may accept a longer wait for better quality. State that budget before optimizing, and keep a correct reference implementation to check against.
There are three ways to improve performance: change the algorithm, change the hardware or numerical representation, and change how the computation uses the machine. They interact. An algorithm with fewer operations can lose because it reads more memory, and a low-precision implementation helps only if the target executes it efficiently at acceptable quality.
Chapter 29.1, Algorithmic speedups starts with reusable structure: separable filters, running sums, pyramids, and approximations. Chapter 29.2, Modern CPUs: memory hierarchy, parallelism, and what it takes to go fast explains memory traffic and parallelism. The three Halide chapters then make loop organization, storage, and recomputation explicit while keeping the algorithm readable: Chapter 29.3, Halide at a high level: decoupling algorithms from schedules, Chapter 29.4, Halide programming, and Chapter 29.5, Scaling up Halide.
The remaining chapters cover Chapter 29.6, Hardware backends: GPU, NPU, DSP, Chapter 29.7, Efficient neural network inference, Chapter 29.8, Automatic search for fast methods, and Chapter 29.9, 8-bit and fixed-point arithmetic. Throughout, compare the complete pipeline at the same output quality and report what the timing includes. A faster kernel matters only to the extent that it improves the workload that uses it.
Performance is part of the method’s usefulness alongside predictable behavior and controllable results (Big lesson L1.1). Keeping the mathematical operation separate from its execution plan lets us ask two different questions: whether the method produces the intended image, and whether this organization uses the machine well. A reference implementation answers the first while schedules, storage choices, and hardware-specific versions explore the second.
Contents of this part
- 29.1 Algorithmic speedups
- 29.1.1 Precompute and look up: pay once, before the loop
- 29.1.2 Separability: a 2-D pass for the price of two 1-D passes
- 29.1.3 Recursive / IIR filters: a running state, cost independent of radius
- 29.1.4 Integral images / summed-area tables: any box sum in four lookups
- 29.1.5 Fast median filters: a sliding histogram for a constant-time median
- 29.1.6 Pyramids and multiscale: do the large-scale work on small images
- 29.1.7 Discretize the range to accelerate non-linear filters
- 29.1.8 Downsampling and edge-aware upsampling
- 29.1.9 Choosing a method
- 29.2 Modern CPUs: memory hierarchy, parallelism, and what it takes to go fast
- 29.2.1 Why moving data, not doing math, is the bottleneck
- 29.2.2 The memory hierarchy and locality
- 29.2.3 The roofline: is my kernel compute- or memory-bound?
- 29.2.4 The forms of parallelism
- 29.2.5 Measuring and improving a kernel
- 29.2.6 Why photography is hard for the machine
- 29.2.7 Keep the comparison valid
- 29.3 Halide at a high level: decoupling algorithms from schedules
- 29.3.1 Images as pure functions
- 29.3.2 The schedule: four questions for every stage
- 29.3.3 What the compiler derives
- 29.3.4 What the split buys, measured
- 29.3.5 How long does a schedule take to write?
- 29.3.6 Where it ended up
- 29.3.7 Letting the compiler find the schedule
- 29.3.8 The split was always there, done by hand
- 29.3.9 Gradient Halide: differentiating the pipeline
- 29.3.10 When to use the split
- 29.4 Halide programming
- 29.4.1 Getting started with Halide
- 29.4.2 Functional, with no loops: Func, Var, Expr
- 29.4.3 A first image pipeline: brighten, then blur
- 29.4.4 Reductions: RDom, sums, and histograms
- 29.4.5 The two extremes: root and inline
- 29.4.6 Scheduling the loops within a stage: reorder, split, tile, vectorize, unroll, parallel, fuse
- 29.4.7 The heart of it: producer-consumer granularity (compute_at, store_at)
- 29.4.8 Comparing schedules with measurements
- 29.4.9 Boundaries and bounds
- 29.4.10 Same algorithm, new machine: the GPU
- 29.4.11 Letting the compiler schedule: the auto-scheduler
- 29.4.12 A repeatable scheduling experiment
- 29.5 Scaling up Halide
- 29.5.1 Decoupling storage from computation: store_at
- 29.5.2 A 7×7 box blur: reductions and update definitions
- 29.5.3 Harris corner detection: ten stages, fifteen edges
- 29.5.4 A brute-force bilateral filter: what a schedule cannot fix
- 29.5.5 The bilateral grid: changing the algorithm instead
- 29.5.6 A three-level Laplacian pyramid: multi-resolution stages
- 29.5.7 The development loop: correctness, measurement, and benchmarking hygiene
- 29.5.8 Going farther: what the tutorials cover that we did not
- 29.6 Hardware backends: GPU, NPU, DSP
- 29.7 Efficient neural network inference
- 29.7.1 Quantization: fewer bits per weight
- 29.7.2 Pruning: fewer weights
- 29.7.3 Knowledge distillation: a small student, a big teacher
- 29.7.4 Low-rank and tensor factorization
- 29.7.5 Efficient architectures: cheapness designed in
- 29.7.6 Neural architecture search: let the machine design it
- 29.7.7 Hardware-aware deployment and the roofline
- 29.7.8 Comparing deployable models
- 29.8 Automatic search for fast methods
- 29.9 8-bit and fixed-point arithmetic