38.4 Visual microphone🔗⧉

When you speak, the pressure waves of your voice travel out through the air and wash over everything in the room. They are weak, but they are not nothing: they set the surface of a bag of chips quivering, they rock a houseplant's leaves by a few thousandths of a millimeter, they ripple the surface of a glass of water. These vibrations are far too small and too fast for the eye, but they are a mechanically filtered response to the sound. The object acts as a diaphragm being driven by the very pressure waves a microphone would catch. The question this chapter answers is startling the first time you hear it: if a camera can see those vibrations, can it recover the sound? The answer is yes. From silent video of a vibrating object, including high-speed recordings, Davis and colleagues Davis et al. 2014 recovered intelligible speech and recognizable music playing in the room, using a passive optical microphone that touches nothing, emits nothing, and works through soundproof glass.

This is the natural sequel to Chapter 38.3, Motion and video magnification. There, we extracted a tiny, sub-pixel motion at each location and amplified it so the eye could see it. Here we extract the same tiny motion, but instead of magnifying it for display, we read it out as a one-dimensional signal over time and play it through a speaker. The object's micro-vibration carries a filtered version of the audio, modulated onto light; recovering the sound is demodulation. Everything that made Eulerian processing work, namely that a small motion is a small, measurable change at a fixed pixel, best read as a change in local phase, is exactly what we need, now pointed at the ear instead of the eye.

38.4.1 From sub-pixel motion to a sound waveform🔗⧉

The pipeline reuses the motion estimate of the last chapter and adds a demodulation step. Point a camera at a vibrating object, say a bag of chips a few meters from a loudspeaker, and process the video as follows.

1. Measure the motion, everywhere, every frame. Decompose each frame with a complex steerable pyramid and read the local phase in each oriented sub-band at each pixel, using the same phase that represented local motion in Chapter 38.3, Motion and video magnification. A sound-induced displacement of a fraction of a pixel shows up as a small change in local phase over time. Working in phase (rather than raw brightness) matters here even more than before: phase change measures displacement without scaling it by image contrast, but its uncertainty increases at low contrast. Reliable texture and adequate exposure remain essential.
2. Pool into one signal per frame. A single pixel's motion is buried in noise, but the whole object is being pushed by the same sound. Combine local motion signals after accounting for orientation and relative phase; naive averaging can cancel regions vibrating in opposite directions. Weight signals by how reliable they are (textured, high-contrast regions report motion more reliably than flat ones), to collapse the millions of noisy per-pixel estimates into a single, far cleaner motion value for that frame. This is the √N averaging of Chapter 13.2, Denoising by averaging doing the heavy lifting: useful motion can be combined coherently, while independent noise partly averages away, so pooling pulls the audio up out of the noise floor.
3. String the frames into time. The sequence of per-frame motion values, sampled at the camera's frame rate, is a one-dimensional time series $a(t)$, a candidate audio waveform. Band-pass it to the audible range, denoise it (a little spectral subtraction), and play it. Out comes the sound that was in the room: speech you can understand, a melody you can name (Figure 38.4.1⛶ fullscreen).

That is the whole idea: measure sub-pixel motion, pool it across the object, read it out over time. And it is, deliberately, the demodulation twin of motion magnification: same front end, different back end.

38.4.2 Bandwidth: high-speed cameras and the rolling-shutter trick🔗⧉

There is one hard constraint, and it is the plain catch: audio is fast. Intelligible speech needs frequencies up to a few kilohertz, and to sample a signal you must sample at least twice its highest frequency (the Chapter 6.4, Sampling, downsampling, and aliasing). A normal video camera runs at 30 or 60 frames per second, which is nowhere near enough; it would alias the audio into mush. So the cleanest results come from a high-speed camera running at thousands of frames per second, which samples the object's motion fast enough to capture the audio band directly. With a few-kilohertz frame rate, the chip-bag recovery is good enough to transcribe the words.

But you do not always need exotic hardware, because an ordinary CMOS sensor hides extra temporal resolution in plain sight: the rolling shutter (Chapter 2.7, Sensors — photosites, CCD vs CMOS). A rolling-shutter camera does not expose the whole frame at once. It reads the sensor out one row at a time, so each row is captured at a slightly later instant than the row above it. Normally this is a defect (it skews fast-moving objects). Here it is a gift: within a single frame, the rows are a stack of time samples taken microseconds apart, so the effective temporal sampling rate is the row rate, not the frame rate, which is hundreds of times faster. By treating each row's apparent displacement as its own time sample, an ordinary consumer camera can be pushed to recover audio it has no business recovering, using within-frame timing for audio recovery. This requires short exposures: each row still integrates over its exposure interval, suppressing fast vibration. Different rows also observe different surface locations, and gaps between frame readouts complicate the sampling. The recovery is cruder than a true high-speed camera's, but it works at all on a commodity sensor (Figure 38.4.2⛶ fullscreen).

38.4.3 How good is the copy? The object's frequency response🔗⧉

The recovered sound is not a perfect copy, and understanding why is the same physics that connects this chapter to vibration modes. The object is a mechanical filter between the sound and the camera: it does not respond equally to all frequencies. A stiff, light object (a chip bag) follows high frequencies well; a heavy or damped object (a brick, a thick plant) sluggishly rolls off the highs and reports mostly the low rumble. Each object has its own frequency response, and its own resonant modes, the frequencies at which it vibrates most eagerly, exactly the modal shapes the last chapter visualized with Chladni patterns. So the recovered audio is the true sound colored by the object's transfer function: some bands are faithful, others are attenuated or rung-out at a resonance.

This cuts two ways. As a limitation, it means the visual microphone is not high fidelity. Intelligibility, not fidelity, is the realistic goal, and the choice of object matters enormously (a bag of chips is a famously good "diaphragm"; a wall is a poor one). As an opportunity, it means the same video that recovers sound also measures the object: its modes and damping constrain material properties together with geometry, boundary conditions, and mass, so analyzing the sound-induced vibrations becomes a way to estimate material properties from ordinary video Davis et al. 2015. The vibration that carries the audio is also a fingerprint of what the object is made of.

38.4.4 The active cousins: laser vibrometry and interferometry🔗⧉

The visual microphone is passive. It uses only the ambient light the object already reflects, which is what makes it remote, covert, and cheap. Its active counterparts reach the same vibrations with active optics, trading hardware for precision. A laser Doppler vibrometer shines a laser on the surface and measures the Doppler shift of the reflected beam: as the surface moves toward or away by even nanometers, it shifts the light's frequency, and that shift reads out the surface velocity directly, the optical analog of Doppler radar. Many laser Doppler vibrometers themselves use interferometry. An interferometer overlays the reflected beam with a reference beam so that sub-wavelength motion shows up as shifting interference fringes, and imaging those fringes over a whole surface maps out the modal shapes, showing which regions are moving and which are nodal, to a precision the passive method cannot touch.

The trade is the recurring one. Active methods (laser vibrometry, interferometry, and the time-of-flight Chapter 38.6, Active non-line-of-sight methods of the neighboring chapter) buy strong, clean signal and exquisite precision at the cost of emitting light and bringing specialized instruments. The passive visual microphone buys remoteness, stealth, and a commodity camera at the cost of being photon-starved, lower-fidelity, and object-dependent. They are the two faces of the same insight that runs through this whole part: the world's vibrations and the world's hidden light are encoded in signals an ordinary camera records and a little processing recovers. A bag of chips behind soundproof glass can supply the vibrating surface of an optical microphone. As with a wall used as an accidental camera, recovery depends on whether the recording preserves enough of the desired signal. The measurement was already present; the computation makes it usable.