36.0 3D AND DEPTH: part introduction⧉
A camera projects the world onto a sensor and, in doing so, discards one number per pixel: how far away each point was. Recovering that number, depth, and from it full 3-D shape, is one of the oldest and most active problems in vision, and it underlies an enormous amount of what modern imaging does, from a phone's portrait-mode blur to autonomous driving to the immersive displays of the next part. The problem is about breaking the projection's ambiguity: a single pixel could have come from a near small thing or a far large one, so to pin down depth you need more constraints, such as a second viewpoint, a moving camera, a known projected pattern, or a strong learned prior about how the world is usually shaped. This part explains where those extra constraints come from and how to invert them.
Two families organize everything here. Geometry recovers depth from correspondence and triangulation: find the same scene point in two or more views, and the rays that pass through its images intersect in space, the principle behind stereo, structure-from-motion, and multi-view reconstruction, whose ideal geometry is exact, while real estimates depend on calibration, image noise, baseline, and scene motion. Priors recover depth from a single image by leaning on regularities the geometry alone cannot supply, such as the fact that skies are far, that faces have a canonical shape, that texture gradients and shading and occlusion carry monocular cues, knowledge that classical rules encoded by hand and that neural networks now absorb from data. The frontier is where the two fuse: neural radiance fields and Gaussian splatting fit a differentiable 3-D scene to many photos (geometry) with representation and optimization biases, and sometimes additional learned priors, and feed-forward models amortize the whole reconstruction into a single network pass. Read the part as a ladder from pure geometry, through single-image priors, to the learned-scene-representation methods that now dominate. The part ends by asking whether "3-D" is even the right target, or a fake task standing in for what we actually want.
Chapter 36.1, Multiple view geometry lays the foundations: projection, epipolar geometry, triangulation. Chapter 36.2, What "depth" means, and where it comes from, disentangles the several things "depth" can mean and the cues that yield it. Chapter 36.3, Monocular depth estimation (one image → depth) and Chapter 36.4, Single-image 3-D: tour into the picture, photo pop-up, 3-D Ken Burns push depth out of a single image, by learned priors and by layered geometry-plus-inpainting. Chapter 36.5, Multi-view 3-D reconstruction: the classic pipeline and Chapter 36.6, Structured light scanning are the geometry workhorses: many passive views, or one active coded projector. Chapter 36.7, Photos → radiance fields and Gaussian splatting (NeRF, 3DGS) and Chapter 36.8, Feed-forward (amortized) 3-D: skip the per-scene optimization are the neural-scene-representation revolution and its speed-ups. Chapter 36.9, Re-photography, applies the machinery to align past and present. And Chapter 36.10, The landscape, and is 3-D a "fake task"? steps back to ask what all this reconstruction is really for.
Contents of this part
- 36.1 Multiple view geometry
- 36.2 What "depth" means, and where it comes from
- 36.3 Monocular depth estimation (one image → depth)
- 36.4 Single-image 3-D: tour into the picture, photo pop-up, 3-D Ken Burns
- 36.5 Multi-view 3-D reconstruction: the classic pipeline
- 36.6 Structured light scanning
- 36.7 Photos → radiance fields and Gaussian splatting (NeRF, 3DGS)
- 36.7.1 Two goals, one diagram
- 36.7.2 Inverse differentiable rendering, and the discontinuity that forced fuzziness
- 36.7.3 NeRF: a scene as a tiny neural network
- 36.7.4 Do we even need the network?
- 36.7.5 3-D Gaussian splatting: the lessons without the deep learning
- 36.7.6 The practical recipe: and what is baked in
- 36.7.7 Relaxing the assumptions: NeRF in the wild
- 36.8 Feed-forward (amortized) 3-D: skip the per-scene optimization
- 36.9 Re-photography
- 36.9.1 Why you cannot just overlay
- 36.9.2 The real-time guidance loop
- 36.9.3 Where it sits
- 36.10 The landscape, and is 3-D a "fake task"?
- 36.10.1 The field spans several design axes
- 36.10.2 Is 3-D a "fake task"?
- 36.10.3 The counterpoint, and the frontier