38.1 Accidental cameras🔗⧉

Stand in a darkened room on a bright day, with the curtains drawn but for a small opening, and look at the wall opposite the window. If your eyes adapt long enough, a faint, upside-down picture of the street outside becomes visible: cars, trees, the building across the road, all inverted and washed out, painted in soft color across the plaster. No one installed a camera. The small opening in the curtain acts as a pinhole, the wall is its screen, and the room is a camera obscura that has been present all the time for as long as there have been rooms with windows. Camera obscuras have a long history, including their use as drawing aids, and the point of this chapter is that it is not a curiosity. Accidental cameras are everywhere. Once you learn to recognize them, a photograph of a suitably illuminated wall may contain a second, hidden image of part of its surroundings, including things outside the picture, around the corner, or behind you.

The unifying idea, due to Torralba and Freeman Torralba & Freeman 2012 (accidental cameras), is that the world is littered with unintended optical elements, such as small apertures and small occluders, that modulate ambient light just enough to leave a recoverable image on whatever surface happens to be watching. Our job is to find that modulation and invert it. We will meet two complementary devices: the accidental pinhole, where a gap passes a thin pencil of rays and forms a (faint, inverted) image directly; and its photographic negative, the accidental pinspeck, where a small occluder, such as a person, a hand, or a passing object, blocks a thin pencil of rays, so that subtracting the blocked image from the unblocked one recovers what that pencil would have carried. We will see the corner of a room, or the edge of a doorway, act as a one-dimensional aperture that resolves a hidden scene by angle. The visible surface records light filtered by the aperture or occluder. In simple geometries this is a convolution that we can try to undo; more generally we invert a spatially varying light-transport operator. Noise, unknown geometry, and missing information usually require regularization.

38.1.1 The accidental pinhole: a window is a camera🔗⧉

Recall Chapter 2.2, Pinhole Image Formation and linear perspective. A point of the scene sends light in all directions; an opaque wall with a single small hole lets through only the one ray from that point that happens to pass through the hole, and that ray lands at a single spot on the screen behind. Do this for every scene point and you get an image. It is inverted because rays cross at the hole, and dim because the hole throws away all but a sliver of each point's light. The hole's size sets the trade every photographer knows from Chapter 5.2, Exposure and capture timing: smaller reduces geometric blur (a tighter pencil of rays per point) but admits less light. Beyond a certain point diffraction makes a smaller hole blurrier again. A large hole is brighter but produces geometric blur shaped by the opening.

Now notice that nothing in that argument required the hole to be designed. Any small bright opening in an otherwise dark enclosure does it. A gap where the curtains don't quite meet. A keyhole. The space under a door. A pinprick of missing paint on a blacked-out window. A compact opening can cast a faint inverted image on the surface opposite; a long slit largely averages the scene along one dimension. The image of the bright scene outside is usually too dim and too low-contrast to notice against the ambient light in the room, but it can be recorded by a camera that can be told to look for it (Figure 38.1.1⛶ fullscreen). The reason we miss it with the naked eye is not that the image is absent but that it sits buried under the room's own diffuse illumination; subtract a reference frame of the room without the outside scene, or simply boost the contrast of the difference, and the street outside emerges on the wall.

This reframes a banal photograph. A picture of someone's living-room wall, shot on a bright day, may carry, in its faint shading, a low-resolution image of the view out the window the camera never saw. The wall is the screen; the window, partly occluded by its own frame and by furniture, is a messy, extended aperture. The image is bad: soft, dim, and geometrically distorted by the irregular opening. But it is still an image of a scene outside the camera's field of view, formed for free.

38.1.2 The accidental pinspeck: the anti-pinhole🔗⧉

The pinhole's mirror image is subtler and, for our purposes, more powerful. Instead of a small hole in an opaque wall, consider a small opaque speck in front of an open, fully lit surface, such as a fleck of dust, a finger, or a person standing in a doorway. Torralba and Freeman call this a pinspeck, or anti-pinhole, camera, and the trick is to think in terms of what the speck subtracts.

The idea is simple. Take a patch of wall lit by a whole region of the scene, many directions of incoming light all summing at that patch, which is why an open surface shows no image: each point of the wall integrates light over many directions, washing out angular detail. Distance, incidence angle, and surface reflectance can still produce spatial shading (this is the Chapter 2.4, Image measurements as integrals over the plenoptic function that destroys the picture). Now place a tiny occluder somewhere between scene and wall. The occluder casts a faint shadow. It blocks exactly the rays from one small direction of the scene from reaching the wall behind it. The difference between the lit wall and the shadowed wall is precisely the light that came from that one blocked direction. A pinhole forms an image by keeping one pencil of rays and discarding the rest; a pinspeck forms an image by discarding one pencil and keeping the rest. You then recover that pencil by subtraction. The speck's shadow is a pinhole image, carried as a small negative dip in brightness, and differencing pulls it out.

This is why the pinspeck is so useful: occluders are everywhere and they move. A person walks across a room; at each instant their body blocks a slightly different slice of the ambient light from reaching the wall. Subtracting a shadowed frame from an unobstructed reference isolates the blocked light. Consecutive frames with a moving occluder instead measure the difference between two shadow patterns: both the old and new occluder positions contribute. Reconstruction must account for that motion, or estimate an unobstructed background. The resulting measurements can reveal parts of the surrounding scene that the camera is not directly pointed at. The occluder that blocks the scene is also what makes recovery possible (Figure 38.1.2⛶ fullscreen).

38.1.3 The occluder as a crude lens, and recovery as deconvolution🔗⧉

Step back and the two devices are one. In both cases an occluder, either the opaque wall around the pinhole or the speck itself, modulates the hidden scene's light before it lands on the visible surface. Write the hidden scene as $H$ (radiance versus direction) and the occluder's visibility function as $A$: $A$ says, for each point of the relay surface and each scene direction, whether the occluder lets that ray through. In general this gives a linear integral operator, with visibility and geometric weights that vary across the surface. Under a planar, approximately shift-invariant model, it simplifies to a convolution:

Here $I$ is measured after subtracting an estimated ambient background, and $n$ includes sensor noise and background-estimation error. Ambient illumination is structured and cannot generally be treated as zero-mean sensor noise. A pinhole makes $A$ a sharp point (sharp image, little light); an open surface makes $A$ flat (lots of light, no image, everything blurred together); a pinspeck makes $A$ "all-but-a-point," whose complement is the point we recover by subtraction; a coded mask would make $A$ a designed pattern. The occluder is a crude, accidental aperture. Recovering $H$ from $I$ is exactly the inverse problem of Chapter 33.3, Coded aperture, with one cruel twist: in a designed coded-aperture camera you know the mask $A$; the accidental occluder's geometry may be unknown and need estimation along with the scene. A measured or deliberately placed occluder can instead give a known operator.

When both are unknown, recovery becomes blind deconvolution in the convolutional approximation, ill-posed in three ways: the scene is unknown, the operator is unknown, and the measurement is noisy. It is precisely here that L7.8show bites. Inverting $A * H = I$ directly amplifies noise into garbage (the same blow-up as naive Chapter 16.4, Blind deblurring); what rescues it is a prior: that natural scenes are smooth in places and sharp at a few edges, that the occluder is a simple shape (a straight edge, a convex silhouette), that the answer should look like a plausible image rather than a field of speckle. With the right prior, a faint smear of penumbra resolves into a recognizable picture. The information was in the light; the prior is what lets us read it out.

38.1.4 Corners and doorways: an edge that resolves the hidden room🔗⧉

The most usable accidental aperture is not a hole at all. It is an edge. Walk up to the outside of an open doorway, or to the corner where two walls meet, and look down at the floor just past the corner. The floor there is lit by the hidden part of the room around the corner, but the wall's vertical edge occludes it: floor points at different azimuths about the corner see different angular portions of the hidden room. The edge acts as a one-dimensional aperture, sorting the hidden scene by angle around the corner, and it writes that sorting into the penumbra, the soft gradient of brightness on the floor between full shadow and full light (Figure 38.1.3⛶ fullscreen).

The penumbra looks like a faint, smooth ramp of shading that the eye dismisses as a shadow. But its gradient is informative: the rate at which the floor brightens as you sweep around the corner encodes how much light is arriving from each successive angle of the hidden room. Differentiate the penumbra with respect to angle and you get a one-dimensional image of the hidden scene versus angle. Because video gives you that penumbra every frame, differencing over time yields a one-dimensional video of motion around the corner: someone walking through the unseen room shows up as a moving bump in the recovered angular signal, even though the camera can see only the floor and the wall. This is the corner camera of Bouman and colleagues Bouman et al. 2017 (corner camera), and it is developed in Chapter 38.5, Corner camera; here it is the cleanest illustration of the chapter's thesis: an ordinary edge, ordinarily lit, is an aperture that images what we cannot directly see.

Push the same idea to two dimensions and you reach computational periscopy Saunders et al. 2019 (periscopy): place a known opaque occluder between a hidden scene and a visible wall, and a single ordinary photograph of that wall, with the occluder's soft cast shadow playing the role of a coded aperture, contains enough structure to deconvolve a full two-dimensional image of the hidden scene. No laser, no time-of-flight hardware, no special sensor: just the penumbra and a solver. These passive methods can use an ordinary camera and ambient illumination, but need enough signal and a suitably informative occluder. Unknown or complicated geometry can make recovery fail. The time-resolved measurements in Chapter 38.6, Active non-line-of-sight provide additional constraints, at the cost of specialized illumination and sensing; those methods also face noise and ill-conditioning.

38.1.5 Where else the world hides a camera🔗⧉

Once the pattern is in view it shows up everywhere. A few cousins round out the family. A glossy eye is a curved mirror that images the scene in front of a person; photograph the eye at high enough resolution and you can recover a (warped, low-resolution) picture of what they were looking at, reflected in the cornea Nishino & Nayar 2004. A shiny coffee, a polished tabletop, a glass of water, and a chrome doorknob can each act as an accidental catadioptric camera, imaging its surroundings onto a surface a normal photograph captures by accident. Even an out-of-focus highlight is the aperture's shape projected onto the sensor, which can reveal the aperture shape. The common thread is the one this chapter began with: light carries far more about the surrounding world than the intended photograph records, and a little optics plus a willingness to deconvolve recovers scenes that were never in the frame.

There is a real limitation to state plainly. Every accidental camera is a bad camera: a smeared aperture, a faint signal, an unknown geometry, and a heavy dose of ambient light and noise. The recovered images are low-resolution, low-contrast, and only as good as the prior that regularizes them; push them too hard and, exactly as in Chapter 14.1, What can super-resolution actually do?, the "detail" that appears is invented, not measured. But that is the right way to see these methods: as a demonstration that the picture is present at all, encoded faintly in light we discard every day, rather than as a way to get a good picture of the hidden scene. The chapters that follow take the same attitude to a different invisible: the motion and sound hiding inside an ordinary video.