BookNotesSlidesP-sets Syllabusdraft · v0.2.49.1
◀▲⌂▶
💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.

36.2 What "depth" means, and where it comes from⧉

The previous chapter showed what a second viewpoint buys you, depth through disparity, and derived the two-view algebra that makes it precise. Before we spend the rest of the part recovering depth from one image, from many images, and from casual photo sets, pin down a question that sounds trivial and is not: what, exactly, is "depth," and where does it come from? Getting the definition wrong is the single most common way to write buggy 3-D code, and the inventory of sources is the map for everything that follows.

36.2.1 Depth means the camera-frame z-coordinate⧉

In this book depth means the $z$-coordinate of a scene point in the camera's frame, its signed distance along the optical axis, measured on the plane parallel to the image sensor. The range, the Euclidean length of the ray from the camera center to the point, is a different quantity. The two agree only for a point dead ahead on the axis; for a point off to the side, the ray is longer than its $z$ (by a factor of $1/\cos\theta$, where $\theta$ is the angle off-axis). A flat wall photographed head-on has constant depth across the whole image but increasing range toward the edges; confuse the two and your reconstructed wall bows toward the camera at the corners.

The reason to standardize on $z$ is that it is exactly the quantity perspective projection threw away. Recall from Chapter 2.2, Pinhole Image Formation and linear perspective that projection maps a 3-D point $(X, Y, Z)$ to image coordinates $x = fX/Z$, $y = fY/Z$. It divides by $Z$ and then forgets $Z$. Unprojection is the inverse operation: given a pixel $(x, y)$ and its depth $Z$, recover the 3-D point $X = xZ/f$, $Y = yZ/f$. Depth is precisely the one missing number per pixel that turns the irreversible projection back into an invertible map. That is why a depth map already is a 3-D reconstruction: pixel grid plus per-pixel $z$ unprojects to a full 3-D point field.

fig-depth-vs-ray-length
Figure 2.2.25 (reproduced). Depth is the z-coordinate, not the ray length. For a scene point off the optical axis, the range (length of the ray from the camera center) exceeds its depth z by a factor 1/cos θ. The two coincide only on the axis. A fronto-parallel wall has constant depth across the image but range that grows toward the edges, the standard pitfall when "depth" is defined loosely.

Two data structures recur throughout the part, and they are the same object at different levels of processing. A depth map is just a per-pixel $z$ image, one scalar per pixel, the same resolution as the photo, often visualized as brightness (near = bright) or a colormap. A point map, the term DUSt3R popularizes later in this part, is its unprojected form: a per-pixel 3-D point $(X, Y, Z)$ in some chosen coordinate frame, i.e. the depth map already pushed out into space through the camera intrinsics. A depth map needs the camera's calibration to become 3-D; a point map has that step already baked in. Both are image-shaped: a depth map stores one scalar per pixel, while a point map stores a 3-D vector, which is exactly why they are so convenient for neural networks to regress.

36.2.2 Where depth comes from: a cue-and-sensor inventory⧉

Depth can be computed from images or measured by hardware, and the methods divide along that line. Each entry below has a proper home (some earlier in the book, some in the chapters ahead), but collecting them on one page makes the contrasts visible. The active sensors (the lower half of the list) are deliberately only pointed to here: this part is about computing depth and shape from images, not about depth-sensing hardware, which lives with the sensors and optics material.

fig-depth-cue-catalogue
Figure 5.8.2 (reproduced). The pictorial depth cues a single flat picture can carry, including occlusion, relative and familiar size, height in the visual field, linear perspective, texture gradient, shading, cast shadow, aerial perspective, and defocus blur, each on a minimal schematic scene, with a sidebar listing the cues a flat print cannot give (binocular disparity, convergence, accommodation, motion parallax). These are exactly the cues a monocular-depth network must learn to read.
fig-active-depth-taxonomy
Figure 34.4.2 (reproduced). The active depth sensors, side by side: structured light (project a known pattern, triangulate like stereo, Z = f·B/d), pulsed time-of-flight / LiDAR (clock a pulse's round trip, Z = c·τ/2), and continuous-wave time-of-flight (read depth off the phase shift of a modulated beam, Z ∝ φ). They all replace a passive cue with controlled, emitted light. Pointed to here for contrast; the structured-light case is the subject of a chapter below.

36.2.3 Relative vs metric: the scale you usually don't have⧉

One catch runs through nearly every method above. A calibrated rigid multi-view reconstruction without a known metric baseline recovers geometry, in a nondegenerate case, only up to a similarity transform: an unknown overall scale, plus a rigid rotation and translation. The shape is right and the relative depths are right (this is nearer than that, by this ratio), but the absolute size of the world is unknown. Two views of a dollhouse and two views of a real house can produce identical reconstructions; nothing in the images themselves says which is which. This is relative (or "up-to-scale") depth.

To get metric depth, in real meters, something must fix the missing scale: a known baseline between two cameras, a calibration target of known size placed in the scene, or a metric sensor (LiDAR, time-of-flight) that measures distance in physical units directly. A single image is more ambiguous still: unconstrained geometry is not determined even up to one affine transformation. Some monocular estimators choose scale-and-shift-invariant depth or inverse depth as their training target; others use metric supervision. Keep the words straight: relative may mean ordering, scale-only depth, or an affine depth/disparity estimate; these preserve different quantities, while metric depth gives distances in physical units. State the convention before combining outputs or evaluating errors.