36.4 Single-image 3-D: tour into the picture, photo pop-up, 3-D Ken Burns⧉
This chapter is about machines synthesizing nearby viewpoints from a single image so that a still photo gains parallax and reads as a small 3-D scene. It is the one-input-view extreme of the novel-view-synthesis spectrum that the rest of the part builds out.
36.4.1 The universal recipe, and why holes are the hard part⧉
Every method here, old or new, follows the same three steps. (1) Estimate coarse geometry for the photo: a depth map, a few planes, or a layered mesh. (2) Warp the photo's pixels to a slightly moved camera, using that geometry to give nearer things more parallax than farther things. (3) Inpaint the disocclusions. Step three is the hard part. The moment the camera moves off the original viewpoint, surfaces that were hidden behind the foreground swing into view. The original photo contains no pixels for them, because the camera never saw them. Naïvely warped, those regions are black gaps, "rubber-sheet" stretches, or smeared edges. Filling them plausibly (generating color and depth for surfaces that were never photographed) is a job for Chapter 21.1, Inpainting, texture synthesis, and it is why single-image 3-D is geometry guess plus generative fill (Figure 36.4.1).
This also explains a representational choice that recurs: a layered scene model (billboards, a pop-up of planes, a layered depth image) beats a single depth map, because a layer can hold what is behind another layer, the very pixels a flat depth map, which stores one depth per pixel, cannot represent. A layered representation can store an inferred hidden surface consistently across views. It does not supply evidence for that surface; the hidden content still comes from assumptions, user input, or inpainting.
36.4.2 Tour Into the Picture: the spidery mesh⧉
The idea is Tour Into the Picture Horry et al. 1997 (Tour Into the Picture). Its insight is that a huge class of photos and paintings (streets, hallways, rooms, landscapes with a horizon) are dominated by a single vanishing point, and around that point the background can be modeled as a simple box: a back wall, plus floor, ceiling, and two side walls receding toward the viewer. Horry, Anjyo, and Arai gave the user a direct tool, the "spidery mesh": drag a rectangle for the back wall and place the vanishing point inside it, and lines from the vanishing point to the four image corners automatically carve the rest of the picture into the four receding planes (Figure 36.4.2). Foreground objects that stand off this box (a person, a pillar, a tree) are pulled onto upright billboards (texture-mapped vertical rectangles standing on the floor plane), masked out from the background by hand. With that coarse geometry in place, the system re-renders the scene from a moving camera, and the still "opens up" into a navigable 3-D space. It is geometry-light, almost entirely manual, and effective for the scenes it fits.
36.4.3 Automatic Photo Pop-up⧉
Automatic Photo Pop-up Hoiem et al. 2005 (Photo Pop-up) removed the user from the loop. Building on geometric-context labeling, it classifies every pixel as ground, vertical, or sky, estimates where the vertical regions meet the ground, and then, like a children's pop-up book, "cuts" along the bottoms of the vertical regions and "folds" them up off the ground plane, perpendicular to it. The sky is dropped, the ground becomes the floor, and the verticals become standing cutouts. The output is a coarse 3-D model with no user input at all, good enough for a convincing fly-through of outdoor scenes. It is the fully automatic descendant of the spidery mesh: the same "background planes plus upright cutouts" idea, with a learned labeler doing the work the user used to do by hand.
36.4.4 3-D Ken Burns and 3-D photos: the monocular form⧉
The modern, fully learned form fuses this lineage with the previous chapter's monocular depth. Run a monocular-depth network on the photo, build a layered depth image or mesh from it, render the fly-through, and inpaint both color and depth behind the foreground so the disocclusions are filled. The classic "Ken Burns effect" pans and zooms across a flat photo; 3-D Ken Burns Niklaus et al. 2019 (3D Ken Burns) does it with real parallax synthesized from estimated depth, so the camera move reveals the method's inferred three-dimensional structure, not newly measured hidden surfaces. The same machinery powers the "3-D photos" that social apps generate from a single picture, and 3-D photo inpainting Shih et al. 2020 (3D Photography) makes the disocclusion fill explicit: it builds a layered depth image and runs a learned inpainter that hallucinates both the missing color and the missing depth in the holes, so the synthesized geometry is consistent as the camera moves. It follows the same recipe, estimate, warp, then inpaint, with learned depth and inpainting coupled to geometric rendering.
From the hand-fitted spidery mesh of 1997 to the learned monocular-depth-plus-inpainting of today, the recipe stayed the same; only the automation changed. And the bottleneck never changed: generating what was never seen.