• the **part theme made into a recipe**: the cheapest way to run an expensive operator is *not to run it at full resolution at all*. **Downsample → run the cheap/slow operator at low resolution → upsample the result.** Most useful image operations (tone curves, exposure/color adjustment, dehazing, depth/matte estimation, a learned enhancement) vary **slowly in space**, so their *output* is low-frequency even when the *image* is not — you lose almost nothing by computing them coarse and paying only a cheap upsample. Cost drops by the downsample factor squared.

• **why naive (bilinear) upsampling is wrong**: a plain bilinear/bicubic upsample is **edge-blind** — it interpolates across object boundaries, so a result computed at $\tfrac{1}{8}$ resolution comes back with **halos and blurred, bleeding edges** wherever the operator's output jumps at an edge the low-res grid could not see. The fix is to **upsample with the full-resolution image as a guide**, so the edges of the *result* snap to the edges of the *image*. [`fig-edge-aware-upsampling`]

• **joint / cross bilateral upsampling** (Kopf, Cohen, Lischinski & Uyttendaele 2007, [@kopf-etal-2007]): upsample the low-res solution with a **bilateral filter whose range weights are read from the high-res guide** — a low-res output pixel contributes to a full-res pixel only if their *guide* colors agree, so smoothing follows the guide's edges and never crosses them. The canonical edge-aware-upsampling primitive; turns "compute coarse, upsample crisp" into one filter.

• **the guided image filter** (He, Sun & Tang 2010/2013, [@he-etal-2010]): a faster, **non-iterative, $O(N)$** alternative that assumes the output is a **local linear transform of the guide**, $q_i = a_k I_i + b_k$ over each window; fitting $(a,b)$ by local ridge regression and averaging gives edge-aware filtering and **guided upsampling** with no bilateral-grid bookkeeping and no gradient-reversal halos. The locally-linear-model view that recurs in the next bullet.

• **the bilateral grid as the data structure** (Chen, Paris & Durand 2007, [@chen-etal-2007]): the $(x,y,I)$ lift from the previous section is exactly the structure that makes edge-aware filtering *and resolution change* fast — **splat** pixels into a coarse grid, **blur** linearly, **slice** back out by trilinear interpolation at each full-res pixel's $(x,y,I)$; slicing *is* an edge-aware upsample. Leads to **bilateral guided upsampling** (Chen, Adams, Wadhwa & Hasinoff 2016, [@chen-etal-2016]): fit a **per-grid-cell affine model** (the guided-filter idea, in the grid) at low resolution, then slice it onto the full-res image — approximate *any* complex, slow pipeline by a grid of local affine fits, evaluated for almost nothing at full res.

• **the learned descendant — HDRnet** (Gharbi, Chen, Barron, Hasinoff & Durand 2017, [@gharbi-etal-2017]): a network predicts the **bilateral-grid of affine coefficients** from a *downsampled* input, and the full-res image is produced by **slicing** those coefficients — so the expensive deep work happens at low resolution and only a cheap, edge-aware, per-pixel affine apply runs at full res, fast enough for real-time on a phone. The end state of the whole idea: *learn the coefficients coarse, slice them crisp.*

• 💡 **the tie-back**: every chapter trick said "do less work"; this one says **do the work at lower resolution and pay only a cheap edge-aware upsample** — the same move as pyramids, but for *any* slow or learned operator, with the full-res image as the guide that keeps edges crisp. (Cross-ref [[Bilateral filtering]], [[Joint bilateral and flash/no-flash]] if present, [[Pyramids and wavelets]].)

• `fig-bilateral-grid` — the bilateral grid's three operations: **splat** (scatter pixels into the coarse $(x,y,I)$ grid), **blur** (a plain linear 3-D convolution on the grid), **slice** (trilinear lookup back at each full-res pixel's position) — the structure behind fast edge-aware filtering and slicing/upsampling. (NEW — schematic.)

• the counterintuitive fact that reorganizes everything: a floating-point **multiply-add is ~1 cycle**; a **main-memory access is ~200 cycles** (and ~100× the energy). So for most image code the processor spends its life **waiting for data**, and "make it faster" means **move fewer bytes**, not do less arithmetic.

• the **memory wall**: compute throughput has grown far faster than memory **bandwidth** and *much* faster than memory **latency** for decades — "latency lags bandwidth lags compute." You can **hide latency** with enough parallel work in flight; you **cannot fake bandwidth** — it's a hard ceiling.

• consequence for imaging: most per-pixel operators (point ops, blurs, demosaicking, resampling) do **little math per pixel** and **touch every pixel once or twice**, so they are **bandwidth-bound** — limited by how fast you can stream the image through, not by the ALUs.

• the **pyramid**: registers (fastest, ~KB) → **L2.6** (~32 KB, ~4 cyc) → **L2.19** (~MB) → **L2.13** (~tens of MB) → **DRAM** (~GB, ~200 cyc) → SSD/network. Each level is ~10× **bigger and slower** than the one above. [`fig-memory-hierarchy`]

• **cache lines & locality**: memory moves in **64-byte lines**, so using **neighboring** data is nearly free (spatial locality) and **reusing** data while it's still cached is free (temporal locality). Random or strided access **wastes most of every line** and stalls on misses.

• 💡 **the highest-leverage transform — tiling/blocking**: restructure the loops so the **working set fits in cache** and every loaded byte is **reused many times** before eviction. A separable blur, a pyramid level, a stencil — all run dramatically faster tiled, purely by improving locality, with the *same* arithmetic. This is the core of what a Halide **schedule** chooses.

• **row-major reality**: images are stored row-major (or planar), so **sweep along rows**, keep the inner loop contiguous, and process in **tiles** that fit L1/L2 — the difference between cache-friendly and cache-hostile code is often **10×** at identical flops.

• one picture tells you what to optimize. Plot attainable **GFLOP/s** against **arithmetic intensity** (FLOPs per byte moved): a **slanted roof** = `bandwidth × intensity` rising until it hits the **flat roof** = peak compute. [`fig-roofline`]

• **low intensity** (a copy, a point op, a 3×3 blur ≈ a few flops/byte) sits under the **bandwidth roof** → **memory-bound**: tuning the math does nothing; **fuse stages and improve locality** to move fewer bytes. **High intensity** (a big matmul/convolution, lots of reuse) reaches the **compute roof** → optimize the arithmetic/vectorization. Knowing which side you're on tells you *which* optimization can possibly help.

• a **ladder** from fine to coarse; real code uses several at once. [`fig-parallelism-ladder`]

• **instruction-level (ILP)** — the CPU **pipelines** and runs multiple independent instructions per cycle (superscalar, out-of-order); mostly automatic, but you help it by giving it independent work and predictable branches.

• **data parallelism / SIMD** — one instruction over a **vector** of lanes (SSE/AVX on x86, NEON on ARM): 8–16 pixels per instruction. The compiler (or Halide's `vectorize`) must map the inner loop onto lanes; this is the **single biggest CPU win** for pixel code.

• **thread / core parallelism** — split the image into **tiles across cores** (and SMT/hyperthreads); near-linear speedup for embarrassingly parallel imaging, bounded by **Amdahl** (the serial fraction) and by memory bandwidth shared across cores.

• **SIMT (the GPU)** — thousands of lightweight threads run in lockstep **warps**; the hardware hides memory latency by **switching warps** rather than caching. Ideal for uniform per-pixel work (→ GPU section).

• **distributed / multi-node** — across machines (and accelerators), for training, huge batches, and offline rendering; dominated by **communication** cost, a different regime.

• **overlap / async** — compute while you transfer: **DMA**, **prefetch**, **double-buffering**, CUDA streams. Keeping the pipeline full is half of real performance.

• 💡 **Amdahl vs Gustafson**: Amdahl — the **serial part** caps the speedup, so a 5%-serial program tops out at 20× no matter the core count; Gustafson — in practice we **grow the problem** (bigger images, more frames) so the parallel part dominates and scaling holds.

• the checklist behind every fast kernel: **locality** (tile to fit cache), **enough parallel work** to hide latency (ILP + threads + GPU **occupancy**), **vectorized** inner loops, and **fused** producer→consumer stages so intermediates stay in registers/cache instead of round-tripping to DRAM (the Halide `compute_at` recompute-vs-locality trade-off).

• the classic **stalls** to avoid: **cache misses** (bad access pattern), **false sharing** (two threads writing the same cache line), **branch divergence** and **uncoalesced memory** on the GPU, and **load imbalance** across cores/warps.

• 💡 **the organizing rule**: the **algorithm** fixes the FLOPs; the **schedule** (loop order, tiling, vectorization, fusion, parallelization) fixes the **data movement** — and on real machines the schedule usually decides the speed. That separation is exactly what [[#Performance engineering and scheduling (Halide)]] makes explicit and retargetable; this chapter is *why* it matters.

• a **GPU** trades the CPU's few big **latency-optimized** cores (deep caches, branch prediction, out-of-order) for **thousands of small throughput-optimized** cores fed by **high-bandwidth memory** — it hides memory latency not with caches but by having **so many threads** that some are always ready to run. [`fig-cpu-vs-gpu`]

• **why imaging loves it**: per-pixel work (filters, pyramids, warps, blending, demosaicking, resampling, NN inference) is **embarrassingly parallel** and **uniform**, mapping straight onto the SIMT model. The GPU is the workhorse for **real-time** preview and video.

• **the programming models**: **CUDA** (NVIDIA), **OpenCL** and **Vulkan compute** (cross-platform), **Metal** (Apple); **Halide targets these backends**, so one algorithm retargets from CPU SIMD to GPU compute by changing the schedule.

• **what makes GPU code fast (or slow)**: **memory coalescing** (adjacent threads read adjacent addresses), **occupancy** (enough warps to hide latency), **shared/local memory** as a programmer-managed cache for tiles, and **avoiding branch divergence** (threads in a warp taking different paths serialize). Same two ideas as the CPU — **locality and parallelism** — with the knobs renamed.

• **when the GPU loses**: small images, serial/branchy algorithms, heavy host↔device **transfer** overhead, or pointer-chasing data structures — the CPU's big caches and branch prediction win there.

• the modern camera is a **heterogeneous system-on-chip**: CPU, GPU, NPU, DSP, ISP, and video codec on one die, sharing memory; a single photo touches **several** of them. [`fig-heterogeneous-soc`]

• **NPU / TPU (neural accelerators)** — Apple **Neural Engine**, Qualcomm **Hexagon** AI, Google **Tensor**/**TPU**: dedicated **matrix-multiply engines** (systolic arrays) that run CNN/transformer inference at **far higher throughput-per-watt** than CPU/GPU — the engine behind Night mode, learned demosaicking/denoising, segmentation, semantic ISP. The optimization problem shifts from cache tiling to **operator fusion, quantization (int8/fp16)**, and **keeping the NPU fed**. (→ [[#Hardware backends: GPU, NPU, DSP]]; models in [[Deep learning]].)

• **DSP (digital signal processor)** — low-power, deadline-driven cores (e.g. Hexagon) for the **real-time 3A** loop (auto-exposure/focus/white-balance) and sensor-coupled stats inside the camera subsystem; a different regime from batch image processing — power and latency over throughput.

• **ISP (image signal processor)** — the camera's **fixed-function** pipeline (demosaic, denoise, tone, sharpen, lens correction) in dedicated silicon: the fastest, lowest-power option *because* it is hard-wired, at the cost of flexibility — the hardware embodiment of the [[Basic image processing and ISP|ISP]] pipeline.

• **FPGA / ASIC** — reconfigurable (FPGA) or custom (ASIC) silicon for the highest performance-per-watt when the algorithm is fixed and the volume justifies it (sensors, codecs, datacenter inference); the end of the **flexibility ↔ efficiency** spectrum opposite the general CPU.

• 💡 the spectrum to carry away: **CPU → GPU → NPU/DSP → ISP/ASIC** trades **flexibility for efficiency**; a real pipeline places each stage on the unit that fits it, and the engineering is **orchestrating the heterogeneous machine**, not optimizing one core.