36.3 Monocular depth estimation (one image → depth)⧉
A human glances at a single photograph and sees depth: the chair is in front of the wall, the road recedes to a point, the mountains are far because they are hazy and bluish. It feels immediate and certain. It is also impossible in the strict sense. Holding both facts at once explains the rest of the chapter.
36.3.1 Why one image cannot determine depth⧉
Monocular depth estimation is the textbook ill-posed inverse problem. The forward map, a 3-D scene projecting to a 2-D image, destroys a dimension, and infinitely many 3-D scenes project to exactly the same picture. A small object up close and a large one far away can paint identical pixels; a flat painted backdrop and the real deep scene it depicts are pixel-for-pixel indistinguishable (the whole art of trompe-l'œil is exactly this non-uniqueness). No amount of cleverness extracts a unique depth from the image alone, because the information is genuinely not there.
What rescues the task is a prior: extra assumptions about which of the infinitely many scenes is plausible. Humans carry a learned prior: we know roads are flat, that hazy means far, that occluding things are nearer, that people are roughly person-sized. A monocular depth estimator is, in the end, a model that encodes that prior. The arc of the field is just the arc of how the prior is built: first hand-coded, then learned by regression, then learned at foundation scale, then borrowed wholesale from a generative image model. The cues the prior keys on are the pictorial depth cues from the previous chapter (occlusion and T-junctions, linear perspective and vanishing points, texture gradient, relative and familiar size, aerial perspective, shading, focus blur), exactly the cues a painter uses to put depth into a flat canvas, here run in reverse to read it back out. (That symmetry is the tie to Chapter 40.1, Human factors and the art of photography: the network is learning the same cues painters use to depict depth.)
36.3.2 Relative depth, and the scale-and-shift ambiguity⧉
A single image does not constrain arbitrary scene geometry even up to two global numbers. Scale-and-shift ambiguity is instead a useful output convention adopted by several learned estimators. Calibrated two-view reconstruction with unknown baseline magnitude has a scale ambiguity; a calibrated stereo rig with a known baseline does not. For a model trained to predict affine-invariant depth, calibration seeks a relation of the form
with an unknown positive scale $a$ and shift $b$. With positive $a$, ordering survives this transformation, but ratios generally do not when $b$ is nonzero. A shift in depth changes reconstructed 3-D shape after unprojection. MiDaS instead operates in disparity (inverse-depth) space: its corresponding relation is $1/d_{\text{metric}}\approx a\,q_{\text{pred}}+b$, where $q_{\text{pred}}$ is the predicted inverse-depth quantity. That is not an affine transformation of depth. Check the model's representation before unprojecting it.
Far from being only a nuisance, this ambiguity is what makes modern monocular-depth training possible. If a model is only ever asked to predict depth up to scale and shift, then it can be trained on a mix of datasets, including laser scans in meters, stereo disparities in pixels, structure-from-motion clouds in arbitrary units, and synthetic renders, by using a scale-and-shift-invariant loss that compares predictions to ground truth after fitting the best $a, b$ per image. The model is never penalized for getting the scale "wrong," because scale was never its job. Applied in inverse-depth space, this is MiDaS's idea, and it is why one network can work across indoor, outdoor, close, and far scenes that no single dataset covers.
The flip side is the rule for using the output: a relative prediction can be useful for tasks that mainly need ordering, such as sorting layers, compositing, and synthetic blur, but if you need meters, you need enough independent metric constraints to fit the missing parameters; one known depth generally cannot determine both $a$ and $b$. Such constraints may come from (a known object size, a dual-pixel or stereo cue, a depth sensor). Metric variants, ZoeDepth, Metric3D, and UniDepth, learn metric depth with suitable supervision and camera modeling, sometimes using supplied or estimated intrinsics. Their meter-valued outputs are still predictions subject to domain error.
36.3.3 From hand-built priors to borrowed diffusion priors⧉
The classical era built the prior by hand. Shape-from-X recovered depth from a single physical cue under strong assumptions: shape-from-shading inverts a reflectance model to turn brightness gradients into surface slopes; shape-from-texture reads the foreshortening of a known texture; depth-from-defocus reads the blur. These work in narrow regimes and break outside them. Make3D Saxena et al. 2009 (Make3D) took the first big learning step in this lineage: cut the image into small planar patches and regress each patch's 3-D orientation and position from local features, then stitch them into a coarse 3-D model, supervised by laser-scanned ground truth. In parallel, geometric-context labeling sorted pixels into ground / vertical / sky, a crude but effective 3-D prior that fed the "photo pop-up" of the next chapter.
The learning era replaced hand-built features with end-to-end networks. Eigen et al. Eigen et al. 2014 (single-image depth CNN) trained the first end-to-end CNN for the task, a coarse global network refined by a finer local one, establishing that a network can regress depth directly from pixels. MiDaS Ranftl et al. 2019 improved cross-dataset generalization with the scale-and-shift-invariant, multi-dataset recipe above; DPT Ranftl et al. 2021 (DPT) swapped the convolutional backbone for a vision transformer, sharpening the maps. Depth Anything Yang et al. 2024 (Depth Anything) pushed to foundation scale, training on an enormous corpus of unlabeled images via pseudo-labels from a teacher model, using the same self-/pseudo-labeling approach that scaled language and vision models, now applied to depth. Another route borrows the prior from a generative model: Marigold Ke et al. 2024 (Marigold) fine-tunes a pretrained image diffusion model (Stable Diffusion) to generate a depth map conditioned on the image, exploiting the scene prior the diffusion model already learned from billions of pictures, which ties directly to Chapter 7.4, Generative AI and diffusion, and a clean illustration that "a depth estimator" and "an image generator" can be the same network pointed at a different output.
36.3.4 What the maps are good for⧉
A monocular depth map, even a merely relative one, is enough for a surprising amount. Synthetic bokeh / portrait mode grades a depth-dependent blur to fake a wide-aperture lens from a phone's tiny one (with a matte from Compositing, segmentation and matting to keep edges crisp). AR occlusion can use it once real and virtual depth are expressed in a compatible coordinate and scale convention, the difference between a creature that sits behind your couch and one that floats unconvincingly in front of it. It drives dolly-zoom and relighting effects, and, as the next chapter shows, it lifts a single photo into a navigable 3-D scene. It even initializes the feed-forward reconstructors at the end of this part: accurate depth simplifies reconstruction, but camera calibration, relative pose, visibility, and scale consistency still remain. For every one of these uses, the relative/metric distinction decides whether the result is merely plausible or actually measured, so the rule from the previous chapter still governs: know which kind of depth you hold before you trust it.