• three models: **ray** (geometry), **wave** (color, diffraction), **particle/photon** (counting → noise) — use whichever fits the question [wave/ray/particle figure]

• side bar — **wave–particle duality, made visible with a liquid analog**: light is *both* wave and particle, which is famously hard to picture. A startling **hydrodynamic analog** makes the marriage tangible: a tiny oil droplet **bouncing** on a vibrating fluid bath becomes a **"walker,"** carried across the surface by its own **pilot wave**, and these walkers reproduce quantum-like behavior (single- and double-slit statistics, quantized orbits, tunnelling). They are *not* a theory of quantum mechanics, but they are a gorgeous classical picture of a particle that is inseparable from its wave. [embed video — Couder & Fort and **John Bush's** (MIT) walking-droplet experiments → [[Video Resources]]] (revisited at diffraction, below)

• wavelength, EM map, spectra

• **Newton's prism experiment** (the historical hook): a prism splits white light into the **spectrum** — white light is a *mixture* of wavelengths, and the *experimentum crucis* (a second prism) shows those pure colors don't split further. [classic diagram]

• a spectrum = power vs wavelength; like an **infinity of channels**, of which a camera (and the eye) keep only three → **metamerism** (Color technology)

• a camera sees nothing that hasn't bounced off something

• **reflection** (angle out = angle in); **refraction** (light slows in glass/water → *refractive index*; **Snell's law** n₁ sinθ₁ = n₂ sinθ₂); **diffraction** (→ Wave effects section below)

• marching-band (wavefront) intuition; fuller lens geometry deferred to Single lenses

• **interactive — a wave simulation of refraction** (`fig-refraction-wave-sim`): the same 2-D FDTD wave solver as the diffraction figure, but with a slower medium (higher index) filling the lower half of the box. An angled plane wave crosses the interface and its wavefronts **bend toward the normal** and **crowd together** (λ→λ/n) — the wave speed drops, so Snell's law becomes a *consequence* of the wave picture (the wave-level version of the marching-band intuition and Fermat's equal-path view in [[#Single lenses: refraction and Snell's law]]). Sliders for the index n₂ and the incidence angle, with an on-figure Snell check. Companion to `fig-diffraction-wave-sim`.

• **light–matter interaction is largely *resonance***: much of what light does to matter — the **index of refraction** itself, and **absorption** (hence material color) — comes from the **resonances** of atoms, molecules, and bound electrons. Light's oscillating field drives those bound charges, which resonate at characteristic frequencies (the **Lorentz-oscillator** picture); **near a resonance** the material **absorbs** strongly, and the **index of refraction varies with wavelength** (**dispersion**) according to how far each wavelength sits from the nearest resonance. This is why different materials transmit/absorb different colors, and why $n(\lambda)$ gives **prisms, rainbows, and chromatic aberration**. Flag forward: dispersion **shapes much of lens design** — achromatic/apochromatic doublets (crown+flint), the fight against chromatic aberration in fast lenses → [[Slopendium/Compound lenses, and aberration correction]]. **Interactive** `fig-lorentz-resonance`: a driven bound electron (mass on a spring) with the absorption peak at ω₀ and the n(ω) normal→anomalous dispersion curve; sliders for driving frequency and damping.

• side bar: **polarization** — light is a *transverse* wave (field ⊥ propagation); **polarization** is the **orientation of that oscillation**. Natural light is **unpolarized** (orientation a rapidly varying mix, *no single direction*); reflection partly polarizes it → a polarizing filter cuts reflections / deepens the sky. Two everyday encounters: (a) **good sunglasses are polarizers** that selectively kill glare — *why* lives in the reflection/BRDF material (specular reflection off dielectrics is partially polarized, maximally at **Brewster's angle**) → [[#The BRDF and the look of materials]], the polarizer filter in [[#Lens filters: polarizers, ND, and graduated ND]], and reflection / specular removal in [[Single-image computational photography]]; (b) **LCD displays emit polarized light** — the liquid-crystal shutter forms an image by **rotating polarization between two crossed polarizers**, which is why a polarizer (or polarized sunglasses) makes some screens go dark at certain angles.

• **why some screens go dark through sunglasses and some don't.** Polarized sunglasses pass only **vertically** polarized light (so they can block the horizontally-polarized glare off roads and water). An LCD emits **linearly polarized** light fixed by its **exit polarizer**; whether the screen survives your sunglasses depends on *that polarizer's orientation*: if it emits **vertical** light it passes straight through, if **horizontal** the glasses **black it out**, and at **45°** it dims. Manufacturers choose the axis — laptops and monitors are usually safe in landscape, but many phones (and especially **car dashboards / GPS**, ATMs, and aircraft seat-back screens) are built with a diagonal or horizontal axis, so they wash out or go black when you tilt your head or rotate the phone 90°. The honest fix is to put a **quarter-wave retarder** *outside* the exit polarizer so the emitted light is **circularly** (not linearly) polarized — then it looks the same through any linear polarizer at any rotation; phones that stay bright through sunglasses generally do this.

• side bar: **why the sky is blue and the sunset orange** — **Rayleigh scattering**: air molecules scatter short (blue) wavelengths far more than long (red) ones (∝ 1/λ⁴). At midday the sunlight takes a *short* path through the atmosphere and the blue it sheds scatters into the sky in every direction → the sky reads **blue**; at sunset the light skims a *long* path, its blue is scattered away before it reaches us, and mostly **orange–red** survives → the sun reddens. (Larger droplets scatter all wavelengths about equally — **Mie** scattering → white haze and clouds.) [atmospheric scattering figure] [Minnaert]

• side bar: **rainbows** (spoiler: it's a caustic) — and a desktop demo. A **single spherical raindrop** bends sunlight three times: **refract** in at the front, **reflect** off the back, **refract** out again; because $n$ depends on wavelength (**dispersion**, above) each color exits at a slightly different angle. Across a whole sky of drops those exit rays **pile up** into a bright **caustic** at the **rainbow angle** — about **42° (red)** down to **~40° (violet)** measured from the **antisolar point** (the shadow of your head) — and *that* accumulation is the rainbow. A **glass sphere** reproduces it on a tabletop: it **stands in for a giant water droplet**, and a bright lamp throws its dispersed caustic onto a card (`fig-rainbow-glass-sphere`).

• **the color order, and the second bow**: the **primary** bow has **one** internal reflection — **red on the outside, violet inside**. A fainter **secondary** bow (~51°) comes from **two** internal reflections, with the colors **reversed**; between them sits **Alexander's dark band** (named for Alexander of Aphrodisias, ~200 AD), darker because one-reflection light emerges only at **≤42°** and two-reflection light only at **≥51°**, so the 42–51° gap gets almost no rainbow light (and the sky *inside* the primary is correspondingly brighter). Same geometry, one extra bounce. Real example: a **double bow over Loch Ness** (`fig-rainbow-loch-ness`, photo) showing both bows and the dark band.

• **dispersion alone is NOT a rainbow**: a prism only *fans* colors; the rainbow's brightness and fixed angle are a **caustic**. Vary the impact parameter and the deviation $D(b)$ of the **one-internal-reflection** ray bottoms out at a **minimum** (~138° → bow at 42°) where rays pile up. Crucially, a ray with **only two refractions and no internal reflection makes no bow** — its $D(b)$ is **monotonic**, no minimum, no pile-up. Dispersion only splits the already-bunched light into the band. Interactive Descartes diagram `fig-rainbow-droplet` (water drop; drag the impact parameter, choose the number of internal reflections, and tick **"zoom near the caustic"** to re-range both the slider and the plot tightly around the bow; the deviation-vs-$b$ plot marks the minimum, and a side panel plots the **ray density** — an integral over impact parameter — whose **spike** at the stationary deviation *is* the bow. Dispersion is always on, so three color bands separate near the caustic).

• **it's an angle, not an object**: the bow is centered on the antisolar point, so it can't be reached and *everyone sees their own* — it is a **caustic of accumulated rays at a particular angle**, not a thing sitting in space (the same "it's a direction" lesson as vanishing points). [Minnaert] [`fig-rainbow-glass-sphere`, real demo photo; `fig-rainbow-droplet`, interactive]

• **scattering & absorption** in media (fog, haze, underwater): single vs. multiple scattering → de-weathering / dehazing; the same physics as the blue sky, at close range

• a **red apple is not red in the dark**: color = **illumination × reflectance**

• reflectance ρ(λ) is the surface's intrinsic, fixed signature; illumination S(λ) is whatever light is present; their per-wavelength product C(λ) reaches the eye

• this multiplicative entanglement is exactly what **white balance** must later undo

• scope caveat: the clean product assumes direction-independent reflectance (matte case); the general directional case is the **BRDF**, next

• side bar: **albedo can appear > 100% (fluorescence cheats)** — real (non-fluorescent) reflectance is **≤ 1 per wavelength** (a surface can't return more light than it receives at that wavelength). But a **fluorescent** material **absorbs** short-wavelength (often **ultraviolet**) light and **re-emits** it at **longer** wavelengths, so in the re-emitted band the measured "reflectance" **exceeds 1** — apparent albedo tops 100%. Examples: **optical brighteners** in laundry detergent and paper ("whiter than white"), **highlighter** ink, **fluorescent safety vests**. The trick is **moving energy between wavelengths**, not creating it.

• the **BRDF** (bidirectional reflectance distribution function): a surface re-emits incoming light into a *distribution* of outgoing directions — radiance toward q = f_r(p→q; n, λ) · incoming; *bidirectional* = depends on both the incoming and outgoing directions. [BRDF diagram]

• **diffuse (Lambertian) vs specular (shiny)**: a matte surface scatters into a broad lobe and looks equally bright from any view (the light it *receives* falls off as cos θ); a shiny one concentrates near the mirror direction, giving a view-dependent highlight; most materials are a blend. [Lambertian sphere vs shiny sphere]

• **side bar — where a material's color actually comes from.** "Color" is not one mechanism; the same red can be made several physically different ways, and they behave differently (some shift with viewing angle, some don't), which is why this belongs with the BRDF:

• **absorption (pigments & dyes)** — the everyday case: molecules/electrons absorb certain wavelengths (the **resonance** picture from *Light and optics*) and what's left is the color. Paint, ink, skin, leaves (chlorophyll), most natural surfaces. **Angle-independent** and the basis of the *illumination × reflectance* model above.

• **structural color (interference & diffraction)** — color from **micro-geometry**, not pigment: **thin-film interference** (soap bubbles, oil slicks, anodized metal, the blue of a Morpho butterfly, peacock feathers), **diffraction gratings** (CD/DVD, holographic foil, some beetles), and **photonic crystals** (opal). The hallmark is **iridescence** — the hue **shifts with viewing/lighting angle** — because it comes from path-length differences, so it is strongly **directional** and a genuinely *spectral* BRDF effect, not a single albedo.

• **scattering** — wavelength-dependent **Rayleigh/Mie/Tyndall** scattering makes color without any pigment: the **blue sky** and blue eyes, the **Tyndall blue** of some bird feathers and milk, the red of a sunset (cross-ref the Rayleigh sidebar in *Reflection, refraction*).

• **metallic reflection** — in metals the **free electrons** set the reflectance; most metals are gray/white (aluminum, silver), but **gold** and **copper** absorb in the blue and so reflect warm — a colored *specular* with a tinted highlight (unlike dielectrics, whose highlight stays the light's color). Matters for shading models (metal vs dielectric is the key axis in PBR).

• **emission & fluorescence** — the surface **makes** light: incandescence/LEDs (emit), and **fluorescence/phosphorescence** which absorb short λ and re-emit longer (the albedo->100% and "fluorescence breaks the BRDF" sidebars).

• takeaway: pigment color is a per-wavelength **albedo** (the simple model), but interference, scattering, and metallic color are **angle- and spectrum-dependent**, which is precisely why a faithful material model is a *spectral, directional* BRDF rather than a single RGB reflectance. [PBR: Pharr, Jakob & Humphreys]

• **light–matter interaction also polarizes the light**: reflection (especially the **specular** component) partially **polarizes** what bounces off a surface — maximally near **Brewster's angle** — so a full account of the BRDF is *polarization-dependent* (a **polarization BRDF / pBRDF**). This is exactly what a **polarizing filter** exploits to cut glare and reflections (→ Photography 101 — Lens filters) and what **shape-from-polarization** and reflection-removal methods use (→ Advanced). [cross-ref the polarization sidebar in *Reflection*]

• **side bar — why good sunglasses are polarizers.** Glare off roads, water, snow, and car hoods is light that reflected off a roughly **horizontal** surface, so (near Brewster's angle) it is strongly **horizontally polarized**. Quality sunglasses are **linear polarizers oriented to pass vertical and block horizontal** light, so they kill that glare specifically — not just dim everything like a plain neutral-density tint. That's the difference between "good" (polarized) and cheap (just dark) sunglasses: you see *into* the water and the wet road instead of at the sheen. The same physics is the camera's circular/linear polarizer, and the same selective extinction is shape-from-polarization's signal. (One catch: polarized glasses can make some **LCD screens / phone displays** go dark at certain angles, because those screens emit polarized light.)

• **try it yourself (how to test your sunglasses).** Hold the glasses up to a glary reflection — a car hood, a puddle, a window, a lake — and **rotate them ~90°**. If the glare **darkens in one orientation and comes back in the other**, they are genuine **polarizers**; if nothing changes, they are just tinted. (That is exactly what `fig-polarizer-sunglasses` shows. A second quick test: look at an LCD screen and tilt your head — polarized lenses go dark at some angles.)

• hint of **global illumination** (light bounces many times between surfaces) — and it is not just a rendering nicety, it is something the photographer actively **exploits** and occasionally **fights**:

• **handy — reflectors and bounce flash**: because light keeps bouncing, you can shape it by adding bounce surfaces. A **reflector** (a white/silver/gold panel) catches the key light and **bounces fill** into the shadows; a **bounce flash** fires at a ceiling or wall so the *whole room* becomes a big, soft secondary source instead of a hard on-axis point. Both are global illumination put to work — softening contrast and wrapping light around the subject (→ Photography 101 — fill & bounce flash; [[#Human factors and the art of photography]]; computational bounce flash in Computational illumination). [photo: someone holding a reflector to bounce fill onto a subject — `fig-reflector-fill`, **shoot our own** unless a CC image turns up]

• **a challenge — unwanted bounced color**: the same bounced light can **hurt** you, because it carries the *color* of whatever it bounced off (color bleeding — the very thing the `fig-global-illumination` figure shows). The classic case is shooting at a **pool**: the strong **blue** light bouncing up off the water casts a cold tint on faces that makes decent **skin tones** genuinely hard to hold (a green lawn or a red wall does the same in its own hue). Knowing the bounce is colored is half the fix — block it, add neutral fill, or correct it later.

• **side bar — fluorescence breaks the BRDF.** A BRDF is **per-wavelength**: same wavelength in, same wavelength out, just redistributed in direction. **Fluorescence** violates this — it **absorbs short-λ (often UV) light and re-emits at *longer* wavelengths** (the **Stokes shift**), moving energy *between* wavelengths. So a fluorescent material is not described by a scalar BRDF per λ but by a **reradiation (bispectral) matrix** — input wavelength → output wavelength. (This is also the cheat behind apparent albedo > 100% — cross-ref, don't re-derive, the albedo sidebar in [[#The color of objects: illumination times reflectance]].) Everywhere once you look: **optical brighteners**, **highlighter** ink, **minerals under UV**, **GFP** in bio-imaging, **banknote security inks** — and it is the working signal in fluorescence **microscopy** and **forensics**.

• **side bar — subsurface scattering (why skin isn't plastic).** A BRDF assumes light leaves at the *same point* it entered. In **translucent** materials light instead **enters, scatters around beneath the surface, and exits somewhere else**, so a point-wise BRDF is **insufficient** — you need the **BSSRDF** (bidirectional *surface-scattering* RDF), which couples an entry point to a separate exit point. Examples: **skin**, **marble**, **milk**, **wax**, **leaves**. This is what gives skin its soft, "lit-from-within" glow and rounded shadow terminator; render skin with a pure BRDF and it looks **plastic**. (Cross-ref translucency / global illumination in rendering.)

• **radiance** L: power per unit area per unit solid angle carried *along a ray* — ultimately what a pixel measures; **conserved along a ray** in vacuum (a wall doesn't get dimmer as you step back, only smaller)

• **irradiance** E: power per unit area *arriving on a surface* (the sensor) — the integral of incoming radiance over the collecting solid angle, which grows with **aperture area**

• the **cos θ** in E = ∫ L cos θ dω is **projected area**: a beam (cylinder of light) of width w striking a surface at angle θ spreads over a larger footprint w/cos θ, so the same power is thinned → E ∝ cos θ (Lambert's cosine law). [cosine / projected-area figure]

• **sidebar — seasons**: the same cosine explains seasons — the Earth's 23.5° tilt makes summer sun strike North America near-vertically (concentrated) while winter sun arrives at a grazing angle (spread out, ~half the energy per area). [seasons figure]

• **units — radiometric vs photometric** (worth pinning down): the quantities above are **radiometric** (raw energy, in **watts**). Weight them by the eye's spectral sensitivity — the **luminous-efficiency curve $V(\lambda)$** — and you get the **photometric** quantities the photographic world actually quotes:

• radiant flux **W** ↔ luminous flux **lumen (lm)**

• irradiance **W/m²** ↔ **illuminance lux** (lm/m²) — light *arriving on* a surface (an incident-meter reading)

• radiance **W/m²/sr** ↔ **luminance cd/m²** (the "**nit**") — light *along a ray* / leaving a surface; this is the perceptual "brightness" of a patch, and what displays, HDR, and the photopic/scotopic scale (Perception) are all quoted in

• intensity **W/sr** ↔ luminous intensity **candela (cd)** — the SI **base** unit (≈ one candle)

• so **cd/m² = lm/(m²·sr)**; for scale, a monitor is ~100–300 cd/m², an HDR display ~1000+, paper in sunlight ~10⁴, and the photopic/scotopic ranges (Perception) live on this same axis

• **exposure** H = irradiance × time → ties straight back to aperture (D/f)² and shutter t (Exposure settings)

• **falloff**: a *point* source dims as **1/r²** (inverse-square), but scene **radiance is distance-invariant** — contrast the two carefully, it confuses people

• this radiometry underlies **HDR** (recovering scene radiance) and **computational illumination**

• (the off-axis **cos⁴** edge falloff / natural vignetting is deferred to **Sensors**, where the pixel irradiance integral and a finite aperture make it meaningful)

• light is a **wave**: at edges and small apertures it **diffracts** — bends and spreads instead of traveling in straight rays

• consequence for imaging: a **finite aperture cannot focus to a perfect point**; even an ideal lens images a point as a small blur, the **Airy disk**, whose size grows with wavelength × f-number

• the **diffraction limit**: the finest resolvable detail is set by wavelength and aperture (Rayleigh ≈ 1.22 λ/D); a **larger** aperture diffracts **less**

• so **stopping down too far softens** the image — aberrations dominate wide open, diffraction dominates stopped down → a sharpness **sweet spot** (developed with MTF in the Optics part)

• **coherent vs incoherent**: we mostly assume **incoherent** imaging (intensities add, stay positive, linear); coherent / wave optics returns for advanced optics (metalenses, lensless, holography)

• **a live demo — the wave equation, solved**: diffraction through a slit is shown as an actual 2-D wave simulation (FDTD), wavefronts bending and spreading past the opening, with the time-averaged intensity pattern alongside — and a companion widget where a *smaller aperture visibly spreads more* (θ ≈ λ/D). [`fig-diffraction-wave-sim` (animated), `fig-diffraction-aperture-size`]

• **the audio parallel** (use it — sound is the everyday 1-D wave): every wave effect here has a **sound** counterpart, and most people have *heard* them. Diffraction — **bass bends around a doorway** (long λ) while treble stays directional, exactly as a small aperture spreads light; plus **interference** and **beats**, **standing waves** in a pipe or room, and the **Doppler** shift. The governing math is shared (a wave equation, Fourier content, λ/D spreading), so acoustics is a reliable intuition pump for the optics — and the same FDTD solver in `fig-diffraction-wave-sim` *is* an acoustic simulation with light's numbers.

• side bar — **diffraction and duality, revisited with walkers**: diffraction is the *wave* story (a slit spreads light by θ ≈ λ/D); the *photon* story is that **individual** photons still arrive one at a time, yet pile up into the same fringe pattern. The **walking-droplet** analog (see [[#Light: rays, waves, and the spectrum]]) dramatizes the marriage: a single bouncing drop passing a slit is steered by its own **pilot wave** into a **diffraction-like** distribution — so you can literally *watch* a "particle" guided by a wave. [reuse the Couder/Bush walker video → [[Video Resources]]]

• pinhole,

• show camera obscure example (some artist)

• **geometry of pinhole imaging**: each scene point's chief ray passes through the pinhole onto the sensor. The physical sensor sits *behind* the pinhole, so the image is **inverted** (upside-down and left-right flipped).

• we instead use a **virtual image plane the same distance in front** of the pinhole — *not physical*, but it keeps the image **right-side up**, so we adopt it throughout (consistent with the camera looking down **+z** → [[Conventions, Notation & Style#Geometry & coordinate systems]]).

• the pinhole→sensor distance is the **focal length f**: a short f gives a **wide** field of view, a long f a **narrow** (telephoto) one — FOV = 2·arctan(sensor / 2f).

• **side bar — DIY pinhole.** A pinhole needs no lens, so it is the easiest real camera to build: punch a clean tiny hole in foil and tape it over the lens mount of a camera body, or over one end of a lightproof box (a shoebox / oatmeal tin) with film or paper at the other — a working **camera obscura**. Smaller holes are sharper but dimmer (diffraction sets the floor), so exposures run long.

• **side bar — the pinhole bag** (credit the idea to **Freeman, Torralba & Isola**'s online vision book, their Figure 5.5, at <https://visionbook.mit.edu/imaging.html>): pull a bag with a single small pinhole over your head, let your eyes dark-adapt, and you see the **upside-down image of the bright world outside projected on the inside of the bag** — pinhole projection made personal, and an unforgettable demo that the inverted image is real.

• exercise: make your own, do in a room

• mention animal eyes

• linear perspective when camera is at origin and along z axis

• **perspective projection equation**: a scene point (X, Y, Z) projects to screen coordinates **x' = f·X/Z** and **y' = f·Y/Z** — each screen coordinate is the world coordinate **divided by the depth Z and multiplied by the focal length f** (the "divide by z" that makes distant things small). [shown both as 2D similar-triangles and in a 3D view]

• 💡 **Big lesson — with the camera at the origin looking down +z, perspective is *mostly a division by depth*.** In that canonical frame the whole of linear (pinhole) perspective is just $x' = f\,X/Z,\ y' = f\,Y/Z$: divide each world coordinate by the depth $Z$, then scale by $f$. Everything else a camera does — sitting somewhere else, pointing some other way, landing on pixels — is bolted on *around* this one step. The **divide-by-depth is the whole of perspective**; the rest is bookkeeping. → register [[Big Lessons]]

• **homogeneous coordinates**: write a 2D point as (x, y, 1) up to scale, so projection, **translation**, and camera motion all become **matrix multiplies**

• **translation becomes a matrix, too.** A shift by $(t_x, t_y)$ is *not* linear on $(x, y)$, but on the homogeneous triple it is an ordinary matrix multiply: $\begin{bmatrix}1&0&t_x\\0&1&t_y\\0&0&1\end{bmatrix}\begin{bmatrix}x\\y\\1\end{bmatrix} = \begin{bmatrix}x+t_x\\y+t_y\\1\end{bmatrix}$, and in 3D the $4\times4$ analog $\begin{bmatrix}I_3&\mathbf{t}\\\mathbf{0}&1\end{bmatrix}$ translates $(X, Y, Z, 1)$. This is *the* reason graphics and vision live in homogeneous coordinates — translation, rotation, projection, and the perspective divide all compose into one chain of matrix multiplies. [translation-matrix in homogeneous coordinates]

• so far the camera has sat at the **origin looking down +z**, where perspective is the clean divide-by-depth above. But a real camera is **somewhere else, pointing some other way** — so before we can use that canonical projection we must re-express the scene **in the camera's own frame**. This is the step that turns "an arbitrary camera" into "the camera at the origin."

• 💡 **moving the camera = applying the *inverse* transform to the world.** Sliding the camera right by $d$ is indistinguishable from sliding the whole **world left** by $d$; rotating the camera one way is rotating the world the other. So we never move the camera *inside* the projection — we move the **world into the camera's frame** by the inverse rigid transform, then project with the fixed origin/+z pinhole. (This is also why "camera-to-world" and "world-to-camera" matrices are inverses — the convention trap below.)

• that world→camera step is a **rotation R then a translation t**, written as one $4\times4$ matrix in homogeneous coordinates, $\begin{bmatrix}R&\mathbf{t}\\\mathbf{0}&1\end{bmatrix}$, carrying a world point $(X, Y, Z, 1)$ into the camera frame — where the canonical divide-by-depth then applies. Translating the camera is changing $\mathbf{t}$; aiming it is changing $R$.

• 💡 **Big lesson — perspective for an *arbitrary* camera is one $4\times4$ matrix followed by a divide by the last coordinate.** Pose the camera (world→camera $4\times4$), project, and map to pixels — every stage is a matrix on the homogeneous point, so the *whole* geometric pipeline composes into **one matrix multiply**, and the only nonlinearity — the perspective divide — *is* the final "divide through by the last coordinate." The arbitrary-camera case is the origin case (divide by depth) wrapped in two more matrices; homogeneous coordinates are exactly what make that wrapping elegant. → register [[Big Lessons]]

• moving camera, matrices, all that good stuff

• **camera intrinsics K** — give them a real treatment, not a bullet: reason first in **normalized image coordinates** (project onto the z=1 plane, independent of sensor size), then map to **pixels** with a second, **affine** matrix carrying the focal lengths fx, fy and the **principal point** (the arbitrary image center). So the full projection **decomposes** as *(pixel affine) · (project at z=1) · [R|t]* = K·[R|t]. This split is also why **cropping an image = using a longer focal length** — you change only the pixel mapping, not the projection. [projection-matrix decomposition figure; crop = focal-length figure]

• 💡 **Big lesson — cropping a photo *is* changing the focal length.** From the projection geometry, cropping touches only the last stage, the pixel mapping $K$: keep a smaller patch of the normalized image and rescale it to fill the frame — exactly what enlarging $f_x, f_y$ does. A 2× crop and a 2× zoom produce the *same* picture (the crop just has fewer pixels). Scaled up to whole sensors, this same fact is the **crop factor** that relates sensor sizes. → register [[Big Lessons]]

• **camera extrinsics [R|t]** — where intrinsics are *inside* the camera, extrinsics are the camera's **pose in the world**: a **rigid 6-DOF** world→camera transform, a rotation **R** (3×3, 3 DOF — orientation) plus a translation **t** (3 DOF — position). Together they place the world into the camera's frame *before* projection, so the **full projection is intrinsics · extrinsics = K·[R|t]** (the [R|t] moves the world into camera coordinates; K projects and maps to pixels). "Moving the camera" is just changing R and t.

• **the convention trap — cam2world vs. world2cam** (the one place to slow down): the matrix [R|t] above maps **world → camera**; its inverse maps **camera → world**. Worse, packages disagree on whether **t is the camera's position** (its center C in world coordinates) or its **negation** in the projection (t = −R·C). Mixing the two silently flips signs / mirrors a reconstruction — always pin down *which direction* a stored pose maps and *what t means* before trusting it.

• this is exactly the machinery that **multiple-view geometry** and **panorama homographies** rely on — relating two cameras is composing their extrinsics; a pure rotation between views collapses [R|t] into a **homography** (→ [[Multiple view geometry]]; Pano & homographies).

• vanishing points

• **what projection preserves vs destroys**: straight **lines → lines** and incidence survive; **angles and lengths do not** (parallels converge to vanishing points, a circle can become an ellipse) — the core fact behind perspective

• Photography, focal length, field of view, various trigomometric ideas, magnification. Difference focal length vs distance and how that changes the background

• the same focal-length-vs-distance distinction governs **portraiture**: facial proportions (and even perceived personality) are set by **camera-to-subject distance**, which is *why* portrait lenses (~85–135 mm) flatter — they let you back up (Perona 2007; full treatment in [[#Lenses and focal length]]).

• sensor size and impact on perspective. Maybe too late?

• sidebar: the (weird) meaning of **"35 mm"** = a **24×36 mm** image frame. The "35 mm" is the *width of the film stock* (including the sprocket-hole margins), not the image; Barnack ran 35 mm cine film sideways, giving a 36 mm × 24 mm frame. Today this size is called **"full frame"**, and smaller sensors are quoted by their **crop factor** relative to it (APS-C ≈ 1.5×, Micro 4/3 ≈ 2×, phone sensors much smaller).

• **depth vs. ray length**: a depth sensor's "depth" is the **z coordinate**, *not* the ray length ‖·‖ to the point — keep them distinct; **unproject** (pixel + depth → 3D point) inverts the projection (RealSense & friends do this live). Depth sensors exist (→ ToF / structured light / stereo). [depth-vs-ray-length / unproject figure]

• a real lens is a piece of glass bounded by **two curved surfaces** (two interfaces)

• light bends at each interface by **refraction** — Snell's law: n₁ sin θ₁ = n₂ sin θ₂

• shown explicitly on a **thick lens** (the two interfaces drawn far apart): the angle of incidence and the angle of refraction at *each* surface, with the indices n₁ (air) and n₂ (glass) marked. [thick-lens Snell figure]

• a curved interface bends a bundle of rays; the two surfaces of a single lens together steer parallel light toward a **focal point**

• **a second view — Fermat / equal optical path length**: a lens focuses because **every** ray from the object to the focus has the *same optical path length* — the longer geometric detour through the thick center is exactly compensated by the slower speed (higher index) of glass there, so all paths arrive **in phase** and interfere constructively. The ray (Snell) picture and this wave (Fermat) picture are two takes on the one fact. [equal-path-length figure]

• 🖱️ **Interactive (web edition):** a live **2-D wave simulation of a thick lens** — a point source on the left radiates circular waves that cross a biconvex glass slab (slower inside, higher index n); because the glass is thickest on the axis it retards the center of each wavefront most, turning the diverging wave into a **converging** one that collapses to an **image point**. This is the Fermat/wave picture made literal: focusing emerges from the wave equation, not from a ray rule. Sliders for the surface curvature R, the index n, and the source distance u, with the focus tracking the **lensmaker's equation** $1/f=(n-1)\,2/R$ and the **imaging equation** $1/v=1/f-1/u$ live on the figure. The lens sits in an opaque mount (aperture stop) so all light is forced through the glass; absorbing (sponge) borders, so nothing reflects off the window edges. Companion to `fig-refraction-wave-sim` (single interface) and `fig-diffraction-wave-sim`. [`fig-thick-lens-wave-sim`; interactive, FDTD]

• **lensmaker's equation** ties focal length to the two radii of curvature and the glass index: 1/f = (n−1)(1/R₁ − 1/R₂)

• this already assumes small angles (paraxial); at large angles the rays do **not** meet at one point → **aberrations** (forward ref: Optics part, where compound lenses correct them)

• eyes have optics too (cornea + lens are the two main refracting elements)

• the **thin lens** as a *simplification / linearization* of the single lens: collapse the two interfaces into one plane, assume small angles and negligible thickness → all bending happens once, at the lens plane (forward ref: Optics part for the accurate thick / compound model)

• precisely, **paraxial = first-order optics**: replace **sin θ ≈ θ** (keep only the linear term). Keeping the *next* term — sin θ ≈ θ − θ³/6 (**third-order optics**) — is exactly what produces the **Seidel aberrations**, so "linearization" and "no aberrations" are the *same* approximation (forward ref: Optics part)

• three-ray construction (parallel→focus, through-center, focus→parallel) locates the image

• **focus equation** 1/f = 1/u + 1/v (object distance u, image distance v); an object at infinity focuses at f; **focusing** = changing v (move the lens or the sensor) to image nearer/farther objects

• **conjugates**: each object distance pairs with one image distance — object at ∞ → image at f; far → just beyond f; close → far behind; at the focal point F → rays exit parallel (image at ∞). [conjugates figure]

• **symmetry & reversibility**: the focus equation $1/f = 1/s_o + 1/s_i$ is **symmetric** in object and image distance, so object and image are **conjugate** — **swap them and the rays retrace the same path** (the reversibility of light paths). Object and image planes are **conjugate planes**, and the magnification follows directly ($m = -s_i/s_o$, so swapping inverts it). This object↔image **reciprocity** recurs throughout optics (e.g. why a lens images either way, and why entrance/exit pupils are conjugates).

• **magnification** m = −v/u (the image is inverted)

• **lensmaker's equation** (thin-lens form) 1/f = (n−1)(1/R₁ − 1/R₂) — connects f to the glass shape (from the single-lens section)

• **f-number** N = f/D sets the aperture (→ exposure, → blur)

• (depth of field gets its **own chapter**, next)

• (placed here, just before **sensors**, because it pays off as the framing for what a pixel actually measures)

• the **plenoptic function** describes *all the light filling space*: the radiance of every **ray**, parameterized by its 3D position, 2D direction, wavelength, and time — a 7-dimensional function [Adelson & Bergen]

• **any** imaging operation is the **measurement of some portion** of this function; the eye and the camera each sample a slice of it

• a conventional camera records only a **2D projection**: each pixel integrates all the rays reaching it, **discarding their direction** — and that discarded *directional* radiance (the plenoptic function restricted to free space, reduced to **4-D**) is exactly the **light field**. Record it instead of integrating it away and you can choose focus, aperture, and viewpoint *after* the shutter — introduced here, developed in [[Light fields and plenoptic cameras]].

• a recurring unifying view: imaging = sampling the plenoptic function

• **the chapter's thesis in one equation — a pixel value is an integral of the [[#The plenoptic function|plenoptic function]]** against a sampling kernel: each photosite *counts photons*, so it reports the **incoming radiance integrated over four physical dimensions**, weighted by the pixel's response. Schematically,

• pixel value ≈ ∫_time ∫_λ ∫_aperture ∫_(pixel area) L(x, y, θ, φ, λ, t) · r(λ) · (geometry) dA dω dλ dt

• i.e. **measurement = integrate the plenoptic function against the pixel's sampling kernel** — and the kernel's *support* in each dimension is exactly a photographic knob. [fig-pixel-integral: thin lens as an ellipse (aperture solid angle), the sensor, one finite pixel; the integral over aperture + pixel area]

• **each integration dimension ↔ a photographic parameter ↔ a downstream effect** (every later sensing topic is "what happens when you change the support of one axis"):

• **sensor footprint (x, y area)** — the pixel integrates radiance over its **finite area** on the sensor, which sets **resolution** and makes a sample a *box-filtered* average rather than a point sample → **sampling & anti-aliasing** (a bigger footprint blurs but suppresses aliasing; pre-filter vs. point-sample lives here).

• **aperture (solid angle of admitted rays)** — the pixel integrates over the **cone of directions** the lens admits, set by the **f-number** N = f/D. Wider aperture = more light *and* a wider integration cone = a shallower **depth of field** (out-of-focus points spread their energy across many pixels → the circle of confusion). The light-vs-blur trade is this single axis. (→ [[#Depth of field]])

• **exposure time** — the pixel integrates over the **shutter interval** t, which sets exposure (H = E·t) and, when the scene or camera moves during t, produces **motion blur** (the width of the temporal kernel). Light-vs-sharpness, the temporal sibling of aperture. (→ Motion / video)

• **wavelength × spectral sensitivity** — the pixel integrates radiance against its **spectral response** r(λ), collapsing a whole spectrum to one number (exactly like a cone, → [[#Color]]). A **color-filter array** gives each photosite a different r(λ) (R/G/B), so color is *spatially multiplexed* and must be reconstructed → **demosaicking** (Bayer).

• **why this framing pays off**: the rest of the sensing story is just *which kernel, how wide, and what it loses* — antialiasing widens the spatial kernel, depth of field is the aperture kernel, motion blur is the time kernel, demosaicking inverts the wavelength multiplexing. Each is also a **source of noise**: integrating fewer photons (small footprint, short time, narrow aperture) means a noisier estimate (→ Noise). Capture is **linear** in radiance, which is why this integral is well-defined and why **raw** values *mean* something physical.

• the **digital-sensor revolution came in two steps**: first **electronic** — turning light into an *analog electronic signal* (the photosite + readout; the era of the CCD and analog electronic / video imaging) — and only **then digital** — putting an **analog-to-digital converter (ADC)** on the path so that signal becomes **numbers** we can compute on. Computational photography lives on that second step; flag the distinction up front.

• a **sensor** is a 2-D array of **photosites** (pixels), each a potential well that **counts photons** → charge → voltage → a digital number; counting photons is what makes capture ~**linear** in radiance (keep this short — the deep sensor zoo is Advanced)

• a **microlens** over each photosite gathers light onto its active area (**fill factor** < 100% because of wiring), and a **color filter** sits under it → the **Bayer** mosaic (forward-ref: Demosaicking). [sensor cross-section figure]

• the scene-radiance → pixel-value mapping is the **camera response curve**: **raw** is ~linear in radiance, while the rendered (JPEG) image applies a non-linear response (≈ gamma + a tone curve — see Linear-vs-gamma in Color technology); **recovering** that response is exactly what HDR merging needs

• **silicon sees IR**: the band gap sits near **1100 nm**, so a photosite keeps counting well into the **near-infrared** — which is exactly why every color camera cements an **infrared-cut ("hot mirror")** filter in front of the sensor (the long-wavelength tails on the spectral-sensitivity curves are trimmed by *that filter*, not by the silicon).

• **make an IR camera**: remove the hot mirror and substitute a filter that **blocks visible** light, and the same sensor images by reflected **near-IR** (`fig-infrared-photography`, author's near-IR panorama on a full-spectrum-converted DSLR).

• **why it looks alien**: chlorophyll reflects near-IR strongly → foliage glows white (the **Wood effect**); the clear sky is blue only because air scatters short wavelengths (**Rayleigh ∝ 1/λ⁴**), so it scatters near-IR far less and collapses to a **deep near-black** — infrared's signature dark sky, strongest away from the sun, throwing white foliage/clouds into relief; skin smooths. Same reflectance underlies **NDVI** remote sensing, art-conservation/forensic imaging, and IR-illuminated security/scientific cameras.

• **the lesson**: the detector's spectral response is set by **physics** (silicon's band gap) and only *trimmed* to the visible by a filter we can choose to remove.

• **CCD vs CMOS**: a **CCD** shuttles charge across the chip in a "bucket brigade" to **one** amplifier at the edge (clean, uniform, but slow and power-hungry, and it reads the whole frame at once → naturally **global** shutter); **CMOS** puts an amplifier (and now the ADC + logic) **at every pixel** and reads in place, **row by row** (fast, cheap, low-power — now utterly dominant). [CCD bucket-brigade vs CMOS read-in-place schematic]

• what CMOS's per-pixel, row-by-row readout **unlocks** (and costs): on-chip ADC, region-of-interest and high-frame-rate readout, on-sensor HDR and PDAF and binning — but, because rows are read at different times, an **electronic rolling shutter** with its motion artifacts (below).

• the pixel value is an **integral over four dimensions** (aperture × pixel area × wavelength × **time**) — the [[#Image measurements as integrals over the plenoptic function]] chapter owns the first three; **this is the time one**. Over the **exposure time** Δt the well **accumulates** photon arrivals: value ∝ ∫ irradiance dt. [photons-accumulating-in-a-well figure]

• **exposure = irradiance × time** (the other half of the exposure triangle with aperture; ISO is gain *after*). **Reciprocity**: ½ the light for 2× the time gives the same exposure — until it doesn't (sensor reciprocity holds far better than film's reciprocity *failure*).

• the time integral is exactly **motion blur**: anything that moves during Δt is averaged along its path → a long exposure smears motion (light trails, silky water), a short one freezes it. The deliberate use and the *removal* of this integral are [[Motion blur, temporal sampling, and resampling]] and deblurring.

• **what ends the integral is the shutter** — which is the rest of this chapter.

• **mechanical shutter** — a physical curtain (or leaf) opens then closes to time the exposure. A focal-plane shutter is **two curtains**: at short times the second starts closing before the first fully opens, so a **slit** sweeps the frame (already a kind of rolling exposure, in hardware). A **leaf shutter** (in the lens) opens from the center → exposes the whole frame at once (global) and **syncs flash at any speed**.

• **electronic shutter** — no moving parts: the sensor **resets** each photosite to start its integration and **reads it out** to end it. Silent, vibration-free, and capable of very short or very high-frame-rate exposures — but on CMOS it is usually **rolling** (below). Many cameras use an **electronic first curtain** (electronic start, mechanical end) to cut shutter shock while keeping a clean finish.

• **global shutter**: every pixel integrates over the **same** time window (all start and stop together), then the frame is read out. No motion distortion. Native to CCD and to special global-shutter CMOS; costs transistors/area per pixel.

• **rolling shutter**: rows are exposed and **read out one after another**, so each row's time window is **shifted** slightly later than the one above. Cheap and standard on CMOS — but rows see the scene at different instants. [row-vs-time diagram: global = a rectangle in (row, time); rolling = a slanted parallelogram]

• the **artifacts** (all from "different rows, different times"): **skew** (a fast pan or a moving car leans over), **wobble / "jello"** (handheld video shimmer), the **spinning-propeller / guitar-string** surreal shapes, and **partial exposure with flash** (a brief flash lights only the rows open at that instant → a **bright band**, which is why flash sync speed exists). [📺 [rolling shutter explained](https://www.youtube.com/watch?v=dNVtMmLlnoE) — Smarter Every Day; `fig-rolling-shutter` interactive: a vertical rod / spinning blade under global vs rolling readout]

• **lighting & flicker banding**: mains-powered and many **LED/PWM** lights **flicker** (100/120 Hz, or fast PWM dimming); under a rolling shutter, rows captured at different phases of that flicker get different brightness → horizontal **banding** (and color banding if RGB phosphors lag). Fixes: match the exposure to the flicker period, **anti-flicker** modes that time the readout, or a global shutter. Ties to video [[Video stabilization and rolling-shutter correction]].

• **correcting rolling shutter** computationally — estimate the per-row time offset and the camera/scene motion, then warp each row back to a common instant → [[Video stabilization and rolling-shutter correction]].

• **off-axis (cos⁴) falloff** still applies once there's a sensor + finite aperture: rays reaching the frame edge arrive obliquely → **cos⁴θ** dimming toward the corners (**natural vignetting**), distinct from optical vignetting (Optics).

• the conventional sensor **couples** all four axes into one number; the **ideal sensor would decouple** them — measure each separately, combine *after* the shot

• a branch of comp-photo **slices/dices the integral** for a-posteriori image formation: **burst imaging** splits **time** (align, then sum → motion-blur control; → [[Multiple exposure imaging]]); **plenoptic/light-field** splits the **aperture** (DoF + focus after the shot; → [[Light fields and plenoptic cameras]]); **hyperspectral** splits **wavelength** (full spectrum/pixel); **polarization** filters at **multiple angles** split a fifth axis (specularities/reflections, glare, reflection separation). Move = *don't integrate yet; record the pieces, decide later*.

• **dual conversion gain (DCG)** — a CMOS pixel can switch the **sense-node capacitance**, changing the conversion gain (µV per electron): **high-conversion-gain** mode (small capacitance → lower read noise, better low light / high ISO) vs **low-conversion-gain** mode (large capacitance → bigger full well, more highlight headroom). Two settings → two **native ISOs** (**dual native ISO**); reading/combining both extends dynamic range → [[Noise, signal-to-noise ratio and dynamic range]].

• **quad-Bayer (Tetracell / quad CFA)** — the CFA groups **2×2 same-color photosites** under one filter tile. Good light: the sensor **remosaics** back to full-resolution Bayer; low light: it **bins** each group into one larger effective pixel (less noise, ¼ resolution). How 48/50/108 MP phone sensors get clean low-light output and in-pixel staggered-exposure HDR. Variant: **3×3 Nonacell**. → [[Slopendium/Demosaicking and joint methods]], [[HDR merging]].

• **dual-pixel and quad-pixel autofocus (on-chip lens, OCL)** — split a photosite into **2 (dual-pixel)** or **4 (quad-pixel, 2×2 OCL)** sub-photodiodes under **one microlens**; each sub-diode sees a different half/quadrant of the lens pupil → **phase-detection autofocus (PDAF) at every pixel** (a stereo baseline inside the pixel); sub-diodes are **summed** for the final image (no light loss). Sensor structure only — the optics/AF algorithm are in [[Autofocus]].