36.10 The landscape, and is 3-D a "fake task"?⧉
We have climbed a ladder: one image to depth, one image to a fly-through, many images to a reconstruction, photos to a radiance field, and finally a single forward pass to 3-D. Step back and see the field at once, then ask a question the field genuinely does not have a settled answer to: do we even need 3-D?
36.10.1 The field spans several design axes⧉
The methods in this part do not fall on a single line from "worse" to "better." They span several design axes, and almost every method is a position in that space rather than a point on a scale. The main axes (Figure 36.10.1):
- Reconstruction vs novel-view synthesis: is the output geometry (a mesh, a point cloud) or just new 2-D images (3-D optional)? The classic pipeline and DUSt3R sit on the reconstruction side; NeRF and 3DGS lean toward NVS.
- Overfit-per-scene vs amortized: is the work redone for each scene (NeRF, 3DGS) or paid once at training and reused (DUSt3R, VGGT, monocular-depth nets)?
- Baked illumination vs relightable: is the lighting frozen into the representation (standard radiance-field and splat models) or separable into materials you can re-light (the frontier)?
- Number of input views: one (monocular, single-image 3-D), a 360° orbit (NeRF/3DGS capture), or large but finite (internet photo collections). Additional informative, compatible views reduce guessing; duplicate views or changing scenes may not.
- Representation: volumetric/fuzzy (NeRF, splats), hard surface (meshes), 4-D ray space (light fields), or none at all (pure image-to-image NVS).
- How much prior is learned: from hand-built geometry (classic SfM) to foundation-scale learned priors (Depth Anything, diffusion-based depth).
- World-space vs camera-space: is the 3-D expressed in a shared frame or a camera frame? This is separate from whether its scale is metric; point maps can use either frame.
The value of the map is diagnostic: when a new system appears, you place it on these axes and immediately know what it trades and what it inherits. There is no single winner because the axes encode different goals: a roboticist wants metric world-space geometry, a film studio wants relightable materials, and a VR app wants real-time NVS. The "best" method is the one that lands where your application needs it.
36.10.2 Is 3-D a "fake task"?⧉
Now the provocation. Through this entire part, explicit 3-D (a depth map, a mesh, a point cloud, or a radiance field) has been treated as the thing to recover. But notice: for novel-view synthesis, 3-D was already optional, a means to an end (new images), not the end itself. That invites an uncomfortable question.
Richard Sutton's Bitter Lesson Sutton 2019 observes that, again and again in AI, the methods that ultimately win are the ones that scale with computation and data, that is, general learning, and that hand-built human knowledge, however clever, tends to be a bottleneck that scaling eventually bulldozes. The companion cautionary tale is the "parable of the parser": for years, natural-language systems were built around explicit syntax trees, because parsing seemed obviously necessary to understand language, until large language models learned to do the end task (translate, answer, summarize) without ever producing an explicit parse. The parse tree, it turned out, was a "fake task": a hand-built intermediate that we found interpretable, but that the learning machine was happy to skip.
The parallel: is explicit 3-D the parse tree of vision? When data and compute are plentiful, perhaps the explicit 3-D reconstruction is an unnecessary intermediate: interpretable to us, but still a detour the model can route around on its way from photos to whatever we actually want (new views, an answer to "what is behind that?", a robot action). The feed-forward methods of the previous chapter are already a step in this direction: they dissolve the explicit SfM chain. A diffusion model that hallucinates a plausible new view of a scene with no 3-D anywhere in the loop is another. On this reading, much of this part is scaffolding we will eventually discard.
36.10.3 The counterpoint, and the frontier⧉
The counterpoint is practical. Real applications keep wanting actual geometry. A robot that must grasp an object, a car that must not hit a pedestrian, a fabrication pipeline that must 3-D-print a part, an AR system that must occlude virtual objects behind real ones: these need geometrically reliable outputs or decisions. Fabrication explicitly requires geometry; robotics and driving may use learned action or occupancy representations without exporting a full mesh. For them, 3-D is not a fake task; it is the literal deliverable, and "it looks right from this angle" is not good enough when a physical action depends on it. (This is exactly the thread Part 39, Adjacent fields and applications picks up, covering robotics, AR, autonomous driving, and fabrication.) The plain answer is that it depends on the task: where the end goal is an image or a decision, explicit 3-D may indeed be optional scaffolding; where the end goal explicitly requires dimensions, surfaces, or clearances, geometric accuracy is part of the deliverable. Whether that geometry must be exposed as an intermediate representation remains a design choice.
The frontier is where this tension is being worked out: pushing to larger scale and longer context (whole buildings, whole videos); going beyond shape and texture to materials and relightability (un-baking the illumination); handling motion and dynamic scenes (4-D, not just 3-D); learning from less supervision; and removing inductive biases (fewer hand-built assumptions, more let-the-data-decide thinking, that is, the Bitter Lesson operationalized). Whether the destination is "explicit 3-D, done better" or "the end task, done directly, with 3-D as an optional byproduct" is, right now, genuinely unknown.