35.7 Dual photography⧉
A camera and a projector are, optically, the same device run in opposite directions. A camera collects light from the scene and sorts it onto a grid of sensor pixels; a projector emits light from a grid of pixels out onto the scene. Point a projector and a camera at the same scene and you have a system whose input is a projector image and whose output is a camera image, and because light, between them, simply propagates through the scene, the relationship is linear for a fixed scene and linear, calibrated source and detector responses. Dual photography (Sen, Chen, Garg, Marschner, Horowitz, Levoy and Lensch 2005) exploits that linearity, plus one physical symmetry, to do something that sounds impossible: produce the photograph the scene would look like from where the projector stands, as if lit by a lamp at the camera, without ever putting a camera at the projector's location.
35.7.1 Light transport as a matrix⧉
Write the projector's pixels as a vector $\mathbf p$ (how bright each projector pixel is) and the camera's pixels as a vector $\mathbf c$ (how bright each sensor pixel ends up). Because propagation is linear, there is a matrix $T$, the light-transport matrix, relating them:
The entry $T_{cp}$ says how much light from projector pixel $p$ ends up at camera pixel $c$, summed over every path it could take through the scene, including the direct bounce and all the interreflections, scattering, and caustics in between. $T$ is enormous (camera-pixels by projector-pixels) but it is a perfectly well-defined object, and in principle you can measure it: light one projector pixel at a time and record the camera image, and you have read off one column of $T$ per projector pixel. (In practice Sen et al. use cleverer multiplexed patterns and adaptive schemes, exploiting that most of $T$ is near-zero, to acquire it without millions of single-pixel shots, but conceptually the point is simple: measure the transport matrix.)
35.7.2 Helmholtz reciprocity: transpose the matrix⧉
The reason to measure the matrix is reciprocity. Helmholtz reciprocity states that light paths are reversible: the transport along a path is reciprocal when source and receiver directions are exchanged with matching radiometric normalization. Raw fractions between finite areas need not be equal without the corresponding area, solid-angle, and sensitivity factors. Ordinary reciprocal reflection obeys this symmetry (the BRDF is reciprocal), and so, therefore, does the entire transport through the scene. In matrix terms this is a compact statement: the transport matrix for the reversed system, with projector and camera swapping roles, is the transpose $T^{\!\top}$ when the source and receiver basis functions and throughput have been normalized consistently. Fluorescence, nonreciprocal media, or a changing scene require a different model.
So consider the dual system, in which the projector is now a virtual camera and the camera is now a virtual light source sitting where the real camera was. Its image $\mathbf c'$ (an image at the projector's pixels) under a virtual illumination $\mathbf p'$ (light emitted from the camera's pixels) is
We already have $T$; transposing it is free. So from a single measured transport matrix we can render the scene from the projector's point of view, illuminated from the camera's point of view. It is a photograph from a camera that physically never existed at that location, complete with the correct shadows and shading that a lamp at the real camera would have produced (Figure 35.7.1). That is dual photography: the real camera's job was only to measure transport; the picture we produce is the dual one, source and sensor exchanged. It can reveal things the real camera could not see directly. For example, parts of the scene hidden from the camera but visible to the projector show up in the dual image, if measurable indirect paths connect those surfaces to the camera. Reciprocity cannot supply a transport coefficient that was never recorded; under direct transport alone, a surface hidden from the real camera is also unlit by the corresponding virtual source.
35.7.3 The unsettling reach: privacy and seeing the unseen⧉
Dual photography quietly dissolves the assumption that you must have a camera at a viewpoint to obtain an image from that viewpoint. Anything that emits or steers light in a controlled way can stand in for the camera's eye, and that has real consequences.
If a photograph from the projector's viewpoint can be synthesized without a camera there, then a controlled light source becomes a kind of latent camera, and the boundary between "illuminating a scene" and "imaging it from the light's side" blurs. This is the privacy concern raised in this line of work (the Yang privacy angle the part outline flags): a display, a projector, or any modulated light interacting with a scene can, in principle, leak an image of the scene as seen from the light, including regions a deliberately placed camera could not, or was not allowed to, see. The same reciprocity that enables dual photography also makes it a surveillance-adjacent capability, and the broader family of looking-around-corners or non-line-of-sight (NLOS) imaging, which recovers hidden scenes from indirect light bounced off a visible wall (Kirmani et al. 2009; Velten et al. 2012; the confocal NLOS of O'Toole et al. 2018), pushes the same idea further: light that has visited a hidden region carries an image of it, and with the transport understood, that image can be read back out. Dual photography is the in-line-of-sight case of recovering hidden information from transported light.
The connecting move for the part is the swap. Where the light stage (Chapter 35.5, Light domes) controlled directional light to measure a reflectance field, and direct/global separation (Chapter 35.4, Separation of Direct and Global Illumination) patterned light to split transport, dual photography measures the full transport matrix and then exchanges the roles of source and sensor. It is the most complete expression here of "the light is a degree of freedom," because once $T$ is known, projector and camera are interchangeable and the scene can be re-photographed from the measured source/receiver positions under illumination representable by those sampled bases.