Computational Photography, an AI-powered Slopendium

▸Part 23 DEPTH OF FIELD⧉

▸23.1 Bokeh, focus stacking, and depth-of-field control⧉

`fig-dof-coc` · depth of field via the circle of confusion: the double cone behind the lens; points whose blur disk stays within the CoC limit c at the sensor are "acceptably sharp" → near/far DoF limits; CoC + hyperfocal H=f²/(N·c) labeled ✅

`fig-dof-raydiagram` · **interactive** 2-D thin-lens depth-of-field ray diagram + plot: set focal length, f-number, subject distance, circle of confusion and sensor size; the diagram shows the focused cone landing on the sensor and a background point spreading into a blur disk; reads off near/far limits, hyperfocal distance H and magnification; "constant magnification" toggle plots total DoF vs focal length (nearly flat — at fixed framing DoF barely depends on focal length). A lighter companion to the 3-D dof sim. ✅

⬜ figure not yet created

pointer to FUNDAMENTALS for the full geometry)

`fig-bokeh-shapes` · bokeh: out-of-focus highlights take the aperture's shape — circular (many blades / wide open) vs polygonal (few blades, stopped down); synthetic defocus-disk fields + aperture insets ✅

`fig-cats-eye-vignetting` · cat's-eye bokeh: off-axis defocus disks clipped into lemon / cat's-eye shapes by optical (mechanical) vignetting toward the frame edge; round at center → clipped at edge, tilted toward center ✅

`fig-focus-stacking` · optics-chapter illustrative figure (07-07 Figure 4): a stepped-focus stack → sharpness selection → all-in-focus composite. Synthetic per-slice defocus on one photo (`sourced/corn-cobs.jpg`, © Frédo Durand) — license-safe. The full real-data treatment lives in part-08 `fig-focalstack-*` ✅

equations

**all imported** from FUNDAMENTALS (circle of confusion, hyperfocal $H=f^2/(Nc)$, near/far limits, DoF scaling). New here: aperture-shape ⇒ defocus-disk shape (the disk is the **image of the aperture stop**)

cat's-eye area loss from off-axis pupil clipping.

23.1.1 The geometry of focus⧉

▸23.1.2 Recap: the geometry of focus (pointer, not re-derivation)⧉

• briefly restate the circle of confusion, the near/far limits, **hyperfocal** $H=f^2/(Nc)$, and the three knobs (smaller aperture, shorter $f$, farther focus all deepen DoF) — then **point to FUNDAMENTALS** for the full similar-triangles derivation and the framing-invariance result. [`fig-dof-coc`]

• restate the load-bearing caveat: **defocus is not a 2-D convolution** (it is depth- and occlusion-dependent) — the reason naive portrait-mode fakes fail and depth-aware/light-field methods are needed (forward-ref Advanced).

▸23.1.3 The bokeh look: shape and structure of the blur⧉

• the out-of-focus highlight is the **image of the aperture stop**, so its **shape = the aperture's shape**: a round iris → round disks, a few straight blades → **polygons** (pentagons, heptagons). [`fig-bokeh-shapes`]

• **disk structure** sets "good" vs. "bad" bokeh: a **bright-edged** disk (from over-corrected spherical aberration) looks **busy/nervous**; a **uniform or soft-edged** disk (under-corrected, or apodized — *Special optics*) looks **creamy**. The *whole* point of a portrait lens is the disk profile, not just resolution.

• **cat's-eye / lemon bokeh**: toward the frame edge the pupil is **clipped by the lens barrel and front/rear elements** (optical vignetting), so off-axis defocus disks are **cut to lemon shapes**, swirling toward the corners; stopping down restores round disks (and reduces corner vignetting). [`fig-cats-eye-vignetting`]

• ties to *Compound lenses* (pupil/vignetting) and *Special optics* (mirror-lens doughnuts, anamorphic ovals, apodizers).

▸23.1.4 Extending DoF: focus stacking⧉

• **focus stacking**: capture a **stack** at stepped focus distances, then composite the **sharpest pixels** from each into one **all-in-focus** image — the standard fix where DoF is hopeless (**macro**, product, landscape near-to-far) ([Stanford focal-stack paper](https://graphics.stanford.edu/papers/focalstack/focalstack.pdf)). [`fig-focus-stacking`]

• requires per-pixel **sharpness selection** + alignment (focus *breathing* changes magnification between shots → must register); a multi-image-fusion problem (cross-ref the fusion/stacking machinery).

• **the sharpness map is depth-from-defocus**: estimating, per pixel, *which slice of the stack is sharpest* is exactly **depth from focus/defocus** (cross-ref *Focus, autofocus*) — the focus position of maximum sharpness corresponds to that pixel's **depth**, so a focal stack yields an all-in-focus image **and** a depth map for free.

• contrast with *increasing* DoF by stopping down — which costs light and eventually **diffraction** softens everything (FUNDAMENTALS): stacking beats the diffraction wall.

▸23.1.5 Controlling and faking DoF⧉

• **shallow on purpose**: open up / step closer / longer lens to *magnify* the subject — recall DoF is set by **magnification**, not focal length per se (FUNDAMENTALS framing-invariance).

• **fake / computational DoF (portrait mode)**: estimate a **depth map** (dual-pixel disparity from *Focus*, stereo, or learned mono-depth) → apply **depth- and occlusion-aware** blur with a chosen disk shape. Honest about why it is hard (hair, edges, transparency) — full treatment forward-referenced to the Advanced computational-camera part; here we only set up *why* a 2-D blur is wrong and *what* a depth map buys.

• **reminder — defocus is NOT a single 2-D convolution**: the blur **kernel size varies with depth** (a far point blurs more than a near one), and **occlusion boundaries break shift-invariance** (a sharp foreground edge bleeds onto a blurred background, not symmetrically) — so no one uniform kernel applies. This is exactly why portrait-mode background blur needs a **depth map** rather than a flat Gaussian (cross-ref the *depth-of-field geometry* recap above and FUNDAMENTALS [[Image formation and linear perspective|Image formation §Depth of field]]).

• **sidebar — coherent imaging (Barbastathis): out-of-depth is *black*, not blurry**: everything above is **incoherent** imaging, where an out-of-focus object still contributes light (a blur disk). George Barbastathis (MIT) notes that **coherent** imaging built around a *prepared* depth can do something qualitatively different — off-depth objects contribute **no light at all**, so they come out **black, not blurry** (incoherent imaging adds positive *intensities*, so every depth contributes; coherent imaging adds *complex amplitudes*, which can interfere destructively and cancel). Reframes DoF as a hard **on/off depth gate** rather than a graceful blur.

---

▸23.2 Fake (synthetic) depth of field⧉

`fig-portrait-mode-pipeline` · fake/portrait-mode DoF pipeline (9.8): all-sharp input → depth/subject map → depth-aware aperture-disk background blur → sharp subject composited on top ✅

`fig-gaussian-vs-bokeh` · the same defocused background two ways — a flat Gaussian (dim, shapeless) vs a real aperture-disk bokeh where clipped highlights bloom into hard-edged disks (blurred in linear light); fake vs convincing ✅

`fig-occlusion-aware-blur` · defocus is not a 2-D convolution — naive whole-image blur bleeds a sharp foreground edge into the background (a halo), vs occlusion-aware rendering that blurs the background up to the subject's silhouette and keeps the subject crisp ✅

`fig-portrait-failure-modes` · a 2×2 gallery of the tells of fake DoF — matting halo (hair), wrong subject blurred, flat shapeless blur, foreground bleed ✅

equations

the **thin-lens blur radius** vs depth — for a pixel at depth $z$ with focal plane at $z_f$, the circle-of-confusion (hence blur-kernel) radius $c(z)\propto \big|\tfrac{1}{z_f}-\tfrac{1}{z}\big|$ (imported from [[Bokeh, focus stacking, and depth-of-field control|depth of field]] / FUNDAMENTALS)

**energy-correct scatter** — each source pixel splats a disk of total weight 1, so brightness is preserved and clipped highlights bloom.

▸23.2.1 Why phones must fake it⧉

• **small sensor ⇒ deep depth of field**: a phone's tiny sensor (and the short focal length that frames the same scene) makes *almost everything* acceptably sharp — DoF scales with sensor size (FUNDAMENTALS). Great for snapshots, but it reads as **flat**; portraits want the subject **isolated** from a soft background. [`fig-portrait-mode-pipeline`]

• so "**portrait mode**" *computes* the shallow-DoF look the optics cannot deliver: estimate **where each pixel is** (or at least *subject vs background*), then blur by an amount that grows with defocus, with a believable aperture-shaped kernel. The computational counterpart to the optical [[Bokeh, focus stacking, and depth-of-field control|depth of field]] of the previous chapter.

▸23.2.2 Where the depth (or subject) comes from⧉

• **dual-pixel disparity**: split-pixel sensors give a tiny stereo baseline within one lens → a per-pixel **defocus/disparity** cue ([[Autofocus]]); the basis of Google's single-camera Portrait mode (Wadhwa et al. 2018).

• **multi-camera stereo**: two phone lenses give a real (small) baseline → a depth map (with the usual occlusion/texture caveats of [[Sparse matching|matching]]).

• **monocular depth nets**: predict depth from **one image** ([[Monocular depth estimation (one image → depth)]]) — no special hardware, but metrically loose.

• **segmentation / matting**: sometimes a full depth map is overkill — a clean **subject mask** ([[Compositing, segmentation and matting]]) suffices to blur "everything that isn't the person," which is what most phones actually ship.

▸23.2.3 From depth to blur — the thin-lens circle⧉

• **map defocus to a blur radius**: pick a (virtual) focal plane, then give each pixel a kernel whose radius grows with its distance from that plane, $c(z)\propto|1/z_f-1/z|$ — the same circle-of-confusion law as real optics ([[Bokeh, focus stacking, and depth-of-field control|depth of field]]). A far point blurs more than a near one; the focal plane stays razor sharp.

• this is the honest part — a thin-lens model the phone can evaluate per pixel. The hard parts are everything around it: the depth's accuracy at edges, and making the blur itself look real.

▸23.2.4 Realistic bokeh — why a Gaussian looks fake⧉

• the giveaway of a cheap fake is a **flat Gaussian** blur: it smears uniformly, has no disk shape, and **dims** bright points instead of letting them bloom. A real out-of-focus highlight is the **image of the aperture** — a hard-edged round (or polygonal) **disk** — and clipped highlights **bloom** into bright disks. [`fig-gaussian-vs-bokeh`]

• **scatter vs gather**: **scatter** (splat an aperture disk from each source pixel, energy-normalized) renders highlight blooming and disk shape correctly but is costly; **gather** (each output pixel averages a depth-dependent neighborhood) is cheap and GPU-friendly but struggles with blooming and depth discontinuities. Production portrait modes pick a point on this trade-off and fake the rest.

▸23.2.5 Occlusion-aware compositing and matting⧉

• **defocus is not a 2-D convolution** (the load-bearing caveat from [[Bokeh, focus stacking, and depth-of-field control|depth of field]]): a sharp **foreground** edge must **not bleed** into the blurred background — the background blurs *up to* the subject's silhouette, and the crisp subject **occludes** it. Naive whole-image blur produces a tell-tale **halo** around the subject. [`fig-occlusion-aware-blur`]

• render in **depth order** (layers), or matte the subject and blur **only** the background, then composite the sharp subject on top. **Disocclusions** — background that should appear behind the subject's softened edge — have **no real data** (a phone has no light field), so they are **inpainted/guessed**; the irreducible gap between fake and true optical DoF.

• the matte itself is the battleground: **hair, fur, foliage, glass** defeat a hard mask, so portrait modes lean on **alpha matting** and still leave artifacts.

▸23.2.6 Failure modes⧉

• the classic tells (worth naming so a reader can spot a fake): **edge halos** from matting errors (wind-blown hair), a **flat / shapeless** blur (Gaussian, no bokeh), the **wrong subject** blurred (segmentation grabs the railing, not the person), and **foreground bleed** (a sharp hand softened into the background). [`fig-portrait-failure-modes`]

• the honest summary: portrait mode is a **depth-estimation + matting + plausible-blur** stack, only as good as its weakest stage — usually the matte at fine edges.

---

▸23.3 Focal stacks and depth of field extension⧉

⬜ figure not yet created

`fig-focalstack-problem` (a macro/close-up scene at one aperture: only a thin slab is sharp, foreground and background mush — "even f/16 isn't enough") fig-focalstack-problem

`fig-focalstack-capture` · a focus stack — several frames of one scene focused at different distances, the sharp slab moving through depth — plus an inset on refocusing the lens vs translating on a rail 🟨

`fig-focalstack-sharpness` · one frame walked through high-pass → square → smooth to produce its per-pixel sharpness map $s_k$ (pipeline layout with arrows + $\LaTeX$ labels) ✅

`fig-focalstack-why-square` · a 1-D edge whose signed band-pass averages to ~0, and how squaring it makes the local average register the presence of detail 🟨

`fig-focalstack-argmax-composite` · the per-pixel argmax selection map (which-frame-won / coarse depth) beside the all-in-focus image, plus a zoom comparing $\gamma=1$ vs $\gamma=4$ weighting ✅

⬜ figure not yet created

exponent-1 vs exponent-4 weighting)

`fig-focalstack-photomontage` · a weighted-sum merge (ghosting and blur at depth edges) vs a graph-cut-plus-Poisson merge (clean seams) of the same focal stack (Agarwala et al.) 🟨

`fig-focalstack-magnification` · a focus-at-infinity frame overlaid with a focus-close frame to show that refocusing changes scale, so frames must be registered before merging 🟨

💡 **Big lesson (L11.2 · Capture the full set, decide later):** instead of committing the focus distance at the instant of exposure, **record a whole stack of focal planes and choose per pixel afterward**. The slogan: *focus stacking is to the focus distance what HDR bracketing is to exposure and what the light-field camera is to the aperture* — defer an irreversible capture decision out of the moment of exposure, paying with **more data** and a **harder reconstruction (a per-pixel merge)** for the payoff of an all-in-focus image no single shot could make. (Registered in [[Big Lessons]] as **L11.2**; its full first-appearance box sits at the light-field / plenoptic introduction — here it recurs on the **focus axis**, the third sibling after HDR's **exposure axis** and panorama's **viewpoint axis**.)

equations

per-pixel **sharpness** $s_k(p)=\sum_{q\in w(p)} \lVert \nabla I_k(q)\rVert^2$ (local **high-frequency energy** — squared band-pass / Laplacian power, then window-summed/Gaussian-smoothed)

**all-in-focus selection** $k^\*(p)=\arg\max_k s_k(p)$, output $I(p)=I_{k^\*(p)}(p)$

**soft weighted composite** $I(p)=\dfrac{\sum_k w_k(p)\,I_k(p)}{\sum_k w_k(p)}$ with $w_k(p)=s_k(p)^\gamma$ (exponent $\gamma\!\approx\!4$ → near-argmax but smoother)

**Photomontage energy** $E(\ell)=\sum_p D_p(\ell_p)+\sum_{p\sim q}V_{pq}(\ell_p,\ell_q)$ minimized by graph cut (data $D$ = $-$sharpness of label $\ell_p$

smoothness $V$ = seam-cost penalizing visible transitions), then **Poisson** reconstruction across the chosen labels.

▸23.3.1 Why limited depth of field is the problem⧉

• **the optics constraint** (recap from [[Optics]] / Image formation): only objects within the **depth of field** image inside an acceptable **circle of confusion**; everything nearer or farther blurs. DoF *grows* with a smaller aperture (bigger $N$) — but a small aperture lets in less light (→ noise) and eventually hits the **diffraction limit**, so sharpness *falls* again. There's a floor: **even at the smallest usable aperture, you often can't get enough DoF.** [`fig-focalstack-problem`]

• **where it bites**: **macro** (bug/flower close-ups), **microscopy**, **product/jewelry**, **landscape** wanting both a near rock and a far peak crisp. These are exactly the cases the optics chapter flags as unwinnable in one shot.

• **the escape — change axis**: don't push the aperture harder; **take many frames at different focus distances** and **merge**, keeping each pixel from the frame where it's sharpest. The thin in-focus slab **sweeps through depth** across the stack, so *every* surface is sharp in *some* frame. [`fig-focalstack-capture`]

• **the analogy that places it in this part** (L11.2): this is the **same move** as the chapter's other multi-shot methods — HDR captures every *exposure* and picks per-pixel; panorama captures every *viewpoint* and stitches; the focal stack captures every *focal plane* and selects. And the **dual** escape is the [[Computational optics and coded imaging#Light field and multi-aperture imaging|light-field camera]]: record all the aperture **rays** and **refocus after capture** — in fact a captured light field can be refocused into a focal stack and then merged exactly as here (Ng 2005).

• **important non-convolution caveat** (from optics): defocus is **not** a single convolution of the image — the blur radius depends on **scene depth**, not output pixel location, and **occlusion** matters at depth edges. That is *why* a naive merge struggles at depth boundaries (next sections) and why the good merge needs coherent seams.

▸23.3.2 A simple algorithm — sharpness = local high-frequency energy, then argmax⧉

*(Preserves the user's WIP "A simple algorithm" — same intent, fleshed.)*

• **the core idea**: a region is **in focus** when it carries **local high-frequency detail** — sharp edges, texture. So measure, per pixel and per frame, how much high-frequency content sits in a small window, and at each output pixel **pick the frame with the most**. [`fig-focalstack-sharpness`]

• **the sharpness measure**: take the **band-pass / high-frequency** component of frame $k$ (a Laplacian, a high-pass, or a band of a pyramid), and accumulate its **energy** in a window $w(p)$:

$$s_k(p)=\sum_{q\in w(p)} \lVert \nabla I_k(q)\rVert^2 \quad(\text{local high-frequency / Laplacian power}).$$

• **why you must *square* before averaging** (the subtle step from the slides): the raw high-frequency signal **swings positive and negative** — around an edge it goes $+\to 0\to-$. Its local average is **~zero by construction** (low frequencies were removed), so you cannot smooth it directly. **Square it** (or take $|\cdot|$) → all-positive → now the local (Gaussian) average correctly reads "there *is* detail here." [`fig-focalstack-why-square`]

• **the selection**: the all-in-focus image is the per-pixel argmax,

$$k^\*(p)=\arg\max_k s_k(p),\qquad I(p)=I_{k^\*(p)}(p).$$

(Aside: $k^\*(p)$ is itself a coarse **depth map** — "which focal plane was sharpest here" — i.e. **shape-from-focus**, Nayar–Nakagawa 1994. Cross-ref depth estimation.)

• **softening it — weighted composite**: a hard per-pixel argmax is noisy and shows seams. Two fixes from the slides, both about **giving more weight to the best frame while staying spatially coherent**:

• **weighted sum** (then **normalize!**): $I(p)=\dfrac{\sum_k w_k(p)\,I_k(p)}{\sum_k w_k(p)}$ with $w_k(p)=s_k(p)$ — smooth, but a blurry frame still leaks in.

• **sharpen the weights with an exponent**: $w_k(p)=s_k(p)^{\gamma}$, $\gamma\approx4$ — pushes toward argmax (the winner dominates) while keeping a smooth blend; "subtle but sharper" than $\gamma=1$. [`fig-focalstack-argmax-composite`]

• **and smooth the selection map**: beyond weighting, **spatially regularize the choice** (smooth $s_k$, or smooth/MRF the label field) so neighbors tend to come from the same frame — avoiding salt-and-pepper switching between frames. This "spatial coherence in the choice of pixels" is exactly what the advanced method makes principled. [`fig-focalstack-argmax-composite`, weight-visualization inset]

▸23.3.3 The more advanced method: Interactive Digital Photomontage (graph-cut + Poisson)⧉

*(Preserves the user's WIP "More advanced method — Aseem's Photomontage".)*

• **what the simple method gets wrong**: weighted blending **mixes** frames near depth edges → residual blur/ghosting and visible transitions where sharpness ties (the non-convolution / occlusion problem). We want a **clean per-pixel choice** that is **spatially coherent** *and* an **invisible blend** at the boundaries where the choice changes.

• **the recipe — Interactive Digital Photomontage** (Agarwala et al. 2004; "all-in-focus" is one of its showcase applications): two stages, both already developed in this part —

1. **Graph-cut label selection** (L6.4, [[Seam optimization]]): assign each pixel a **source-frame label** $\ell_p$ by minimizing a discrete energy

$$E(\ell)=\sum_p D_p(\ell_p)+\sum_{p\sim q}V_{pq}(\ell_p,\ell_q),$$

where the **data term** $D_p$ prefers the **sharpest** frame at $p$ (e.g. $D_p(\ell)=-s_\ell(p)$) and the **smoothness term** $V_{pq}$ is a **seam cost** that penalizes a label change where the two frames *disagree* (so cuts hide along matching, low-contrast boundaries). Solved globally by **min-cut / max-flow**. This is precisely the seam-optimization energy — the **energy is the design**, the optimizer is off-the-shelf.

2. **Gradient-domain (Poisson) blend** (L6.3, [[Poisson image editing]]): even the best label map leaves small exposure/color/level mismatches across a seam. Copy each region's **gradients**, then **reconstruct** the image whose gradients best match (solve $\nabla^2 f=\operatorname{div}\mathbf v$). The absolute-level jumps vanish; only the (correct) gradients remain → a **seamless** all-in-focus composite. [`fig-focalstack-photomontage`]

• **why this is the same two lessons twice**: *cut* where it's cheap (L6.4 graph cut) then *reconstruct* in the gradient domain (L6.3 Poisson) — the identical **cut-then-reconstruct compose** used for panorama blending and object compositing in this part. Focal-stack all-in-focus is just that pipeline with **"sharpest frame"** as the data term.

• **scale** (from the slides): real macro montages stack **tens** of frames (e.g. 55) — the graph cut handles many labels; commercial tools (**Helicon Focus**, **Zerene Stacker**, Photoshop **Auto-Blend Layers**) implement essentially this.

▸23.3.4 Capturing the stack — hardware, and the magnification trap⧉

• **two ways to sweep focus** [`fig-focalstack-capture`, inset]:

• **refocus the lens** (change the focus distance, camera fixed) — the natural way; what an autofocus motor or a tunable lens does.

• **translate the camera** along the optical axis (a **focus rail**, e.g. CogniSys **StackShot**) — common in macro, where moving the whole camera a few mm is easier and more repeatable than refocusing.

• **the warning — refocusing changes magnification** (the slides' "swept under the rug"): focusing **closer** moves the lens away from the sensor, which **changes the field of view / magnification** (magnification $\sim f/D$). So frames in a focus sweep are **not pixel-aligned** — a focus-at-infinity frame and a focus-close frame show the scene at slightly **different scales**. [`fig-focalstack-magnification`]

• **consequence — register before you merge**: you must **scale/register** the frames (estimate a per-frame scale, often near-pure zoom about the optical axis) **before** computing sharpness and selecting — otherwise the all-in-focus merge stitches mis-registered detail and you get double edges. Translating on a rail has a related but different geometry (perspective/parallax shift), and may even **change viewpoint** slightly (the slides note "here we change the viewpoint as well"), needing a fuller alignment. This is the same **align-then-merge** discipline as HDR and panorama in this part ([[#Image alignment]]).

• **single-exposure alternatives** (pointers): **scanning / sweep-during-exposure** macro rigs integrate the stack optically in one exposure; and the **dual escape** — a [[Computational optics and coded imaging#Light field and multi-aperture imaging|light-field camera]] captures all focal planes in **one** shot and refocuses computationally (Ng 2005), which can then be merged exactly as above. Time-/light-efficient capture of focal stacks is analyzed by Hasinoff & Kutulakos 2009.