Computational Photography, an AI-powered Slopendium

▸Part 24 ILLUMINATION AND REFLECTION⧉

⬜ figure not yet created

`fig-intrinsic-decomp` (one photo → its **reflectance** layer (flat albedo) × **shading** layer (smooth illumination, no texture)) fig-intrinsic-decomp

`fig-retinex-thresholding` · Retinex: threshold the log-gradient (small steps = shading, large steps = reflectance edges), then re-integrate 🟨

⬜ figure not yet created

`fig-multi-illuminant-wb` (a scene lit warm tungsten on one side, cool window light on the other → one global white balance leaves *one* half wrong fig-multi-illuminant-wb

`fig-hdr-merge-antelope` · the HDR merge on a REAL bracket (Antelope Canyon): two exposures (short/long) + their per-pixel reliability weight maps → reliability-weighted radiance average → tone-mapped result; both sun and shadows readable ✅

`fig-reflection-removal-cues` · cues that separate a window reflection from the transmitted scene — ghosting/double-image, polarization, focus difference 🟨

`fig-dichromatic-model` · the dichromatic reflection model: observed color = a body (diffuse) component + a specular (illuminant-colored) component — a plane in color space 🟨

A single photograph mixes things the eye effortlessly keeps apart: it instantly reads a white shirt in shadow as *white-and-shadowed*, not as a gray shirt. The camera can't — it records the **product**, $I = R\cdot S$, illumination times surface (**L2.6**). This chapter is the project of **un-mixing** that product from one image — separating light from surface, or one layer of light from another. And it is the same story as super-resolution and deblurring: the un-mixing is **under-determined**, so a **prior** does the work (**L5.4**). The unifying frame is the **intrinsic-image decomposition**; everything else here is a special case of it. (These are the *single-image* methods; their easier *multi-image* cousins — flash/no-flash, a rotating polarizer, a light stage — live in **Computational illumination (Advanced)**.)

💡 **Big lesson (recurrence of L2.6 + L5.4):** because the world is **multiplicative** — what you measure is light **×** surface — recovering *either factor* from one image is **ill-posed**: a dark patch can be a dim surface in bright light or a bright surface in dim light, and the pixel can't tell you which. So separating illumination from reflectance (or a reflection from a transmission, a shadow from a stain, a highlight from a color) always needs a **prior** about how surfaces, light, and natural images behave — not a tuning knob but the load-bearing part of the method. (→ see **L2.6** · multiplicative world, FUNDAMENTALS; → see **L5.4** · the prior is not optional, [[#Super-resolution]].)

equations

**image as product** $I(x) = R(x)\,S(x)$ — log makes it additive $\log I = \log R + \log S$ (**L2.6/L2.19**)

**Retinex / gradient classification** — assign $\nabla(\log I)$ to $\nabla R$ if $|\nabla\log I| > \tau$ (sharp = reflectance) else to $\nabla S$ (smooth = shading), then **Poisson-integrate** each layer (**L6.3**)

**layer superposition (reflection)** $I = T + R$ (transmitted + reflected), under-determined → priors

**dichromatic model** $I(x) = m_d(x)\,\Lambda_{\text{body}} + m_s(x)\,\Lambda_{\text{illum}}$ (diffuse in the **surface** color direction + specular in the **illuminant** color direction — two color vectors, separable per pixel)

▸24.1.1 Intrinsic images: the unifying frame ($I = R\cdot S$)⧉

• **the decomposition**: split one image into two **intrinsic layers** — **reflectance** $R$ (the surface's albedo, independent of lighting) and **shading** $S$ (the illumination, independent of surface). Per channel $I = R\cdot S$; in **log** it's additive, $\log I = \log R + \log S$ (**L2.19** — why illumination work lives in log/linear, not gamma). [`fig-intrinsic-decomp`]

• **why it's the umbrella**: once you can name "this variation is *light*, that is *surface*," every effect here is an edit on one layer — **white balance** corrects the *color* of $S$; **shadow removal** flattens a dark region of $S$; **highlight removal** strips a specular add-on; **reflection removal** un-sums two whole $I$'s.

• **why it's hard**: infinitely many $(R,S)$ multiply to the same $I$ — the multiplicative ill-posedness in the box. You need a prior on *which* variations are reflectance and which are shading.

• **Retinex** (Land & McCann 1971) — the **classic gradient prior**: assume **reflectance changes are sharp** (a painted edge) while **shading changes are smooth** (light falls off gradually). Threshold the (log-)gradient: **big jumps → reflectance**, **gentle ramps → shading**; then **Poisson-integrate** each (**L6.3**). Crude but captures the whole intuition: *edges are surface, ramps are light.* [`fig-retinex-thresholding`]

• **SIRFS** (Barron & Malik 2015 — *queue ref*): jointly recover **Shape, Illumination, Reflectance** by **MAP** with hand-built priors (reflectance piecewise-constant/sparse, shape smooth, illumination low-frequency). Same data-fit + prior recipe; learned successors (CNNs predicting $R,S$) in [[Deep learning]] (**L5.3**).

• **the takeaway**: intrinsic decomposition is *the* generalization; the rest is "I care about one layer, and I have a cue that pins it down."

▸24.1.2 Multiple-light / spatially-varying white balance⧉

• **the assumption that breaks**: ordinary white balance estimates **one** illuminant for the whole frame and **divides** it out (**L2.6**, [[Measuring and encoding color|Color technology]]) — right only under a *single* color of light.

• **the failure**: real scenes mix lights — **warm tungsten** vs **cool daylight** through a window; flash + ambient; sun + open-shade blue. One global correction neutralizes *one* region and casts the other: fix the tungsten side and the window goes blue. [`fig-multi-illuminant-wb`]

• **the fix — per-pixel light mixture** (Hsu et al. 2008): model each pixel as a **blend of two illuminant colors**, $I(x) = (\alpha(x)L_1 + (1-\alpha(x))L_2)\cdot R(x)$, solve for the **mixing map** $\alpha(x)$ — a **spatially-varying** white balance — then divide out the *local* light. The two colors come from the user, clipped highlights, or clustering.

• **why it's intrinsic-image**: recovering the *color* of the shading layer as it varies — a chrominance-only intrinsic split; the map $\alpha(x)$ is **smooth** (light varies gently) — the Retinex prior on illuminant *color*.

• **encoding note**: divide in **linear light** (mixing/dividing illuminants is linear in radiance — **L2.19**).

• **cross-ref**: the *easy* multi-image version (flash/no-flash, the flash's known color disambiguates) is in **Computational illumination (Advanced)**; learned single-image in [[Deep learning]].

▸24.1.3 Reflection removal — pulling apart a transmission and a glass reflection⧉

• **the problem**: shoot through a window → **two images summed**, the **transmitted** scene ($T$) plus a **reflection** ($R$): $I = T + R$ (this one is **additive**, unlike $R\cdot S$; **L2.19**). Pulling them apart from one $I$ is hopelessly under-determined — pure prior territory (**L5.4**).

• **the cue zoo** (each a different prior breaking the tie): [`fig-reflection-removal-cues`]

• **defocus / blur**: you focus on the *transmitted* scene → the **reflection is out of focus** (weak, broad gradients). Assign sharp gradients to $T$, soft to $R$ (Fattal 2008 — a Retinex-like split on *focus*).

• **ghosting**: a *thick* pane reflects twice → the reflection appears **doubled/shifted**; that known double-impulse lets you identify and subtract $R$.

• **sparse-gradient prior** (Levin & Weiss 2007 — *queue ref*): both layers have **sparse, heavy-tailed gradients**; the sum has *too many* edges → find the $(T,R)$ split **minimizing total edges** (optionally with user scribbles). Same sparse-gradient prior as blind deblurring.

• **polarization**: light off glass is **partially polarized** (Brewster) → a polarizer attenuates $R$ differently from $T$. *(The genuine multi-image method — rotate a polarizer, two frames — lives in **Computational illumination (Advanced)**; single-image methods try to get this without the extra shot.)*

• **learned** (Zhang et al. 2018 — *queue ref*): a CNN trained on synthesized $T+R$ pairs predicts $T$ with a **perceptual loss** so the layer *looks* like a real photo (**L5.3**).

• **the takeaway**: "$I = T + R$ has no answer; pick the cue (focus, ghosting, edge-sparsity, polarization, a learned prior)." A cousin of intrinsic images — un-summing two illuminations.

▸24.1.4 Shadow detection and shadow removal⧉

• **what a shadow *is*, intrinsically**: a region where the **shading** $S$ drops (less light) while **reflectance** $R$ is **unchanged** — same paint, just darker. So shadow removal is an **edit on the shading layer only**: flatten the darkening, keep the texture. This is why it lives under intrinsic images.

• **the detection cue (the crux)**: tell a **shadow edge** (shading discontinuity) from a **material edge** (reflectance): a shadow edge changes **brightness but not hue much** (same surface color), is often **soft** (penumbra) and geometrically smooth; shadow regions are lit by **skylight** (bluer) not direct sun (warmer) — a color tell.

• **removal as a gradient-domain edit** (**L6.3**): label the shadow *boundary* gradients, **zero them**, and **Poisson-reconstruct** — interior texture (small gradients) preserved, the big shadow step removed. Same machinery as gradient-domain compositing. Refine the matte with the **fast bilateral solver** (Barron 2017) so the correction follows the real penumbra (**L4.21**).

• **the penumbra subtlety**: a hard binary mask leaves a dark halo; you need a **soft shadow matte** and to **scale** (multiply the shadowed radiance back up by the recovered shading ratio — a multiplicative correction in linear light, **L2.6**).

• **cross-ref**: the easy multi-image version (**flash/no-flash**) is in **Computational illumination (Advanced)**; learned shadow removal in [[Deep learning]].

▸24.1.5 Specular-highlight removal / "fake polarization" (the dichromatic model)⧉

• **the problem**: glossy surfaces (skin, leaves, plastic, wet roads) show **specular highlights** — the light source reflected, carrying the *illuminant's* color, not the surface's; they blow out texture and fool color/material estimation. We want the **matte (diffuse-only)** image.

• **the dichromatic reflection model** (Shafer 1985 — *queue ref*): reflected light is the **sum of two terms** with **different colors** — a **diffuse/body** term in the **surface's** color + a **specular/interface** term in the **illuminant's** color: $I(x) = m_d(x)\Lambda_{\text{body}} + m_s(x)\Lambda_{\text{illum}}$. The falloffs vary per pixel, but there are only **two color directions**.

• **the geometric picture (why separable)**: plot a glossy patch's pixels in RGB → an **"L"/dog-leg**: a **line** along the body color (matte shading) with a **spur** toward the **illuminant white** (the specular ramp). Identify the two directions (Lee 1979; Klinker–Shafer–Kanade 1988 — *queue ref*), then **project each pixel back onto the diffuse line**. [`fig-dichromatic-model`]

• **"fake polarization"**: a **real** specular killer is a **polarizing filter** (specular reflection is polarized); doing the *same job* purely **computationally from one image** via the dichromatic model is the "fake polarization" — the polarizer's matte look **without** the filter or extra shot. (Lee 1979 also used the highlight's illuminant color for **color constancy** — a highlight is a free white reference.)

• **why it's an intrinsic-image cousin**: the specular term is an **additive contamination of the shading layer** in the *illuminant's* color; removing it recovers the diffuse $R\cdot S$. A **color-direction** prior rather than a gradient prior — different cue, same un-mixing goal.

• **cross-ref**: the genuine optical version (rotate a polarizer) and multi-light specular removal are in **Computational illumination (Advanced)**; learned separation in [[Deep learning]]. **Encoding**: separate in **linear light** so the dichromatic *sum* is physically additive (**L2.19**).

▸24.2 Mixed-lighting white balance⧉

24.2.1 Why a global gain must fail⧉

24.2.2 Estimating the per-pixel mixture: an under-determined inverse problem⧉

24.2.3 Correcting each region for its own light⧉

24.2.4 What came after⧉

▸24.3 Dehazing⧉

⬜ figure not yet created

`fig-dark-channel` (hazy photo, its dark channel, recovered transmission map, dehazed result) fig-dark-channel

Haze isn't a deblurring problem — light is **scattered**, not convolved — but it's the same *idea* as the previous chapter in a new costume: an image-formation model that is **under-determined**, made solvable by **one well-chosen statistical prior**.

equations

**haze image model** $Y(x)=t(x)\,X(x) + A\,(1-t(x))$ (transmission $t=e^{-\beta d}$, airlight $A$, depth $d$, scattering coeff $\beta$)

**dark channel** $X^{\text{dark}}(x)=\min_c \min_{x'\in\Omega(x)} X_c(x')$

▸24.3.1 Dehazing as a prior-driven inverse problem⧉

• the shape: an under-determined image-formation model + a statistical prior that makes it solvable — the **same thesis** as blind deblurring, even though the physics (atmospheric scattering, not blur) is different.

• the (physical) forward model: $Y(x) = t(x)\,X(x) + A\,(1-t(x))$ — each pixel is the clean radiance $X$ attenuated by **transmission** $t(x)$ (how much light survives the haze; $t = e^{-\beta d}$, $d$ = scene depth), **blended with airlight** $A$ (the bright, scattered atmospheric light). Haze = a depth-dependent fade toward $A$.

• why it's under-determined: per pixel we know $Y$ (3 numbers) but want $X$ (3), $t$ (1), and the global $A$ — more unknowns than measurements. Need a prior. [fig-dark-channel]

• the **dark-channel prior** (He, Sun & Tang 2009): in a haze-free outdoor patch, *some* color channel is **nearly zero** somewhere (shadows, colorful/dark surfaces) — the patch's $\min_c\min_{\text{window}} X_c \approx 0$. Haze *lifts* this dark channel (adds airlight everywhere), so **how bright the dark channel is ≈ how much haze ≈ $1-t$.** That single observation pins down the transmission map.

• the pipeline (state as steps): estimate $A$ from the haziest (brightest-dark-channel) pixels → estimate a coarse $t(x)$ from the dark channel → **refine the transmission's edges** so it follows real object boundaries (a matting / guided-filter / edge-aware solve — He's original used soft matting, later the **guided filter**, [[Edge-preserving techniques]]) → invert the model for $X$. [fig-dark-channel]

• the takeaway to land: *the prior is doing all the work.* The model is unsolvable as stated; one well-chosen statistical regularity (dark channel) turns it into a clean recovery — the **same moral as Wiener and blind deblurring**, just a different prior.

• xref: the refinement step ties into [[Edge-preserving techniques]] (guided filtering); the learned successors live in [[Deep learning]].

▸24.3.2 The wider lineage: scattering physics, learned dehazing, and beyond⧉

• **before the dark channel — physics from scattering.** **Cozman & Krotkov** ([@cozman-krotkov-1997]) turned the haze model around to *recover depth from scattering* — given the atmosphere, the depth-dependent fade *is* a depth cue (haze carries information, it doesn't only destroy it). **Narasimhan & Nayar**'s vision-and-weather work ([@narasimhan-nayar-2002]) laid down the systematic physics of imaging through fog, haze, and rain and how to invert it (classically from two images or a known airlight) — the physical foundation the single-image priors later shortcut.

• **learned dehazing.** Replace the hand-built prior with a network. **Zhang, Sindagi & Patel** ([@zhang-sindagi-patel-2019]) train a single network that **jointly estimates the transmission map and produces the dehazed image** end-to-end, instead of estimating $t$ by a prior and inverting separately — the learned successor to the dark-channel pipeline (forward-ref [[Deep learning]]). (Also flagged: **SDDE** — *queue exact ref / author to confirm*.)

• **out of the air, into the water.** **Sea-thru** ([@akkaynak-treibitz-2019]) corrects **underwater** images with the *right* physics: unlike atmospheric haze, water attenuates each wavelength differently and the veiling light varies with range, so a single global airlight is simply wrong — Sea-thru estimates wideband, range-dependent attenuation (from a known depth/range map) and removes the water's color cast where naive dehazing fails.

• **haze as a task, not just a nuisance.** **Sakaridis, Dai & Van Gool** ([@sakaridis-etal-2018]) synthesize fog onto clear images to build training data for **semantic scene understanding in fog**, so dehazing becomes a means to robust *perception* (driving, robotics) — closing back to the depth-from-scattering view that fog *carries* scene information.

▸24.3.3 Differentiable image pipelines and algorithm optimization (Halide)⧉

• the meta-point that closes the inverse-problem run: every method across these two chapters is *an optimization with a prior* — but who **optimizes the pipeline itself**? When a restoration/ISP pipeline has knobs (filter weights, kernel parameters, the whole sequence of stages), you want to **tune them by gradient descent against a quality loss** — which means the pipeline must be **differentiable** and **fast**.

• **Halide** (Ragan-Kelley et al. 2013): a language that **separates *what* an image pipeline computes (the algorithm) from *how* it runs (the schedule)** — tiling, vectorization, parallelism, fusion. One algorithm, many schedules → portable high performance without rewriting the math. This is *the* tool the rest of computational photography is implemented in.

• **differentiable Halide** (gradient Halide, Li et al. 2018): add **automatic differentiation** through the pipeline → you can **backprop a loss to the pipeline's parameters** (or use the pipeline as a layer inside a network). Now "tune the deblurring/dehazing/ISP parameters" or "learn the filter" is just gradient descent through a fast, scheduled pipeline.

• why it closes the chapter: it's the **engine** under the whole "data-fit + prior, minimized" program — and the bridge to [[Machine learning]] (differentiable pipelines = the boundary where classical optimization meets learned components; cf. FlexISP, Heide 2014, as a differentiable-ISP example).

• scope note: keep this **conceptual** (separation of algorithm/schedule; differentiability as an enabler), not a Halide tutorial — point to the language for hands-on use.

▸24.4 Intrinsic images with time lapse⧉

⬜ figure not yet created

`fig-intrinsic-timelapse-stack` (a fixed-camera day sequence: shadows sweep across a constant scene fig-intrinsic-timelapse-stack

`fig-intrinsic-weiss-median` · noisy per-frame log-gradient maps with moving shadow edges, their per-pixel median over time giving clean reflectance gradients, then a Poisson integration to a flat-lit reflectance plus residual shading (Weiss) 🟨

💡 **Big lesson (L11.2 · Capture the full set, decide later — time/illumination axis):** one image can't separate **reflectance** from **shading** (the split is ambiguous — L5.4). A **time-lapse stack under changing light** breaks the tie: record the **whole sequence** and decide per-pixel afterward — **what stays constant is reflectance, what varies is shading**. Same defer-the-decision move as HDR (exposure), focal stacks (focus), hyperspectral (wavelength) and light fields (aperture) — here the deferred axis is **time / illumination**. (→ see Big lesson **L11.2**, [[Big Lessons]]; and **L5.4**, the prior-not-optional ambiguity this resolves.)

equations

image model $I(x,y,t)=R(x,y)\cdot S(x,y,t)$ (reflectance constant in $t$, shading varies)

in **log** $\log I = \log R + \log S$ (product → sum, L2.6/L2.19)

spatial gradient $\nabla\log I(t)=\nabla\log R+\nabla\log S(t)$

**Weiss estimator** $\widehat{\nabla\log R}(x,y)=\operatorname{median}_t\,\nabla\log I(x,y,t)$ (shading gradients vary/cancel, reflectance gradient persists)

recover $\log R$ by **gradient-domain reconstruction** (Poisson, $\nabla^2\log R=\operatorname{div}\,\widehat{\nabla\log R}$)

then $\log S(t)=\log I(t)-\log R$.

▸24.4.1 The split, and why one image can't do it⧉

• **the model**: an image is approximately **reflectance × shading**, $I=R\cdot S$ — albedo (the surface's intrinsic color/lightness, **fixed**) times illumination/shading (how much light reaches each point, **scene/lighting-dependent**). Separating them is the classic **intrinsic-images** problem (Barrow & Tenenbaum 1978): recover the "paint" from the "light."

• **why it's ill-posed from one frame** (L5.4): infinitely many $(R,S)$ pairs multiply to the **same** $I$ — is a dark patch dark *paint* or *shadow*? Single-image methods need a **prior** (Retinex's assumption: **large** gradients are reflectance edges, **smooth/gentle** gradients are shading — threshold the gradient field, then integrate; the gradient-domain cousin in [[Poisson image editing]]). A prior is **not optional** here.

• **the time-lapse escape (L11.2)**: fix the camera and shoot a **day-long sequence**. The **scene's reflectance is constant**; only the **illumination/shading moves** (sun arc, shadows sweeping). So the ambiguity disappears in the stack: **constant-in-time ⇒ reflectance; varying-in-time ⇒ shading.** No hand prior needed — the *temporal* dimension supplies the disambiguation, exactly as extra focal planes or exposures supplied it for the other stacks. [`fig-intrinsic-timelapse-stack`]

▸24.4.2 Weiss 2001 — the median of log-gradients⧉

• **go to log** (L2.6/L2.19): $\log I = \log R + \log S$ turns the product into a **sum**, so gradients are additive: $\nabla\log I(t)=\nabla\log R + \nabla\log S(t)$. (Work in linear light first, then log — the encoding lesson.)

• **the key statistic — median over time**: at each pixel, the **reflectance** gradient $\nabla\log R$ is the **same in every frame**, while the **shading** gradient $\nabla\log S(t)$ **moves** (shadow edges sweep through, sometimes here, sometimes gone). A **moving-shadow edge appears in only a minority of frames** → it is an **outlier**. So take the **per-pixel median across time** of the log-gradient:

$$\widehat{\nabla\log R}(x,y)=\operatorname{median}_t\,\nabla\log I(x,y,t).$$

The median **rejects** the transient shading gradients and keeps the **persistent** reflectance gradient — a one-line robust estimator. [`fig-intrinsic-weiss-median`]

• **reconstruct (L6.3)**: integrate the estimated reflectance gradient field back to an image — a **gradient-domain / Poisson** solve, $\nabla^2\log R=\operatorname{div}\,\widehat{\nabla\log R}$ — yielding the **flat-lit reflectance** image. Then the per-frame **shading** falls out by subtraction: $\log S(t)=\log I(t)-\log R$.

• **why this is elegant**: it is **L2.6/L2.19** (log makes ×→+), **robust statistics** (median kills outlier shading edges), **L6.3** (integrate gradients) and **L11.2** (the temporal stack disambiguates) in four lines — and it needs **no learned prior**, in contrast to single-image intrinsic decomposition.

24.4.3 Passive and controlled illumination⧉

▸24.4.4 Where this chapter belongs — passive vs. active illumination⧉

• **Recommendation: keep this chapter HERE, in *Multiple exposure imaging*, as the brief time/illumination sibling — and plant a one-line forward-pointer to *Computational illumination*.** Rationale:

• The organizing principle of *this* part is **L11.2 — capture a stack along one axis, decide later**: HDR (exposure), panorama (viewpoint), focal stacks (focus), hyperspectral (wavelength). Time-lapse intrinsic images is the **time/illumination** axis — it belongs to the **same family** and reinforces the spine. Its machinery (log, gradients, **median**, Poisson reconstruction) reuses tools this part already teaches.

• It is fundamentally about **merging a passively-captured set of frames** — the camera is **fixed** and you exploit illumination that **happens** to vary. That is multi-exposure-imaging in spirit.

• **Computational illumination** is about **actively controlling** the light (flash/no-flash, multi-flash, photometric stereo, structured light). Weiss is the **passive** counterpart: same $I=R\cdot S$ goal, but you don't *set* the lighting, you *observe* it change. So note the kinship and **forward-ref** computational illumination (where the active versions live), but the **passive time-lapse** belongs here.

• Resolves the WIP's "??? Or in computational illumination?": **here**, brief, with an explicit cross-reference to computational illumination for the active-lighting decompositions.

▸24.5 Colorization⧉

24.5.1 One channel in, three channels out⧉

24.5.2 A spectrum of priors: scribbles, references, and learned models⧉

24.5.3 The multimodality trap: why naïve colorization goes muddy⧉

24.5.4 Closing the loop: learned priors with a human's hints⧉

24.5.5 Plausible is not correct⧉

24.5.6 Where it sits⧉