38.2 Reflections in the eye⧉
In any sharp portrait, a close crop of the eye shows a reflection of the surrounding scene. The window, the photographer, the soft-box or the bald sky, the room they are standing in: all of it is there, curved and shrunk, painted across the glossy front of the eye. Portrait photographers have a name for the brightest piece of it, the catchlight, and they fuss over its shape and placement because it is the difference between a living gaze and a dead one. But the catchlight is only the loudest note. The visible cornea reflects a broad portion of the surroundings, and the catchlight is the bright part that stands out against iris texture, noise, and other contamination. The thesis of this chapter is that the rest of that reflection is recoverable, and that a photograph of an eye is therefore a limited-resolution, wide-angle sample of the scene in front of the person, a scene that is otherwise entirely outside the frame.
The reason is purely optical. The front of the eye, the cornea, is a smooth, wet, convex surface, close enough to a section of a sphere that we can model it as one. A convex mirror images a very wide field of view onto a small area; this is exactly why the convex mirrors in shops and on blind corners let you see a whole aisle or a whole junction at a glance. The cornea is a miniature version of the same device, permanently aimed at whatever its owner faces. Pair it with the camera that took the portrait, a mirror feeding a lens, and you have a textbook catadioptric system (catoptric, mirrors, plus dioptric, lenses). The eye is the most literal accidental camera in this part of the book: a curved reflector that bounces the surrounding world straight into the photographer's sensor, rather than a hole that lets light through like the Chapter 38.1, Accidental cameras. Reading it out reveals the scene behind the camera, in front of the subject, that the picture was never pointed at.
38.2.1 The cornea as a catadioptric mirror⧉
What makes the eye usable, where a flat cheek or a matte wall is not, is specularity plus curvature. The corneal surface is covered by the tear film, an optically smooth layer that reflects like polished glass: a few percent of incident light bounces off in the mirror direction rather than scattering diffusely. That specular component is what carries an image. A diffuse (Lambertian) surface throws every incoming ray off in all directions and so blends the whole scene into a flat wash, because it records the light but destroys the picture, the same integration that makes an open wall useless in Chapter 38.1, Accidental cameras. A mirror preserves the angular structure: the direction a ray leaves the surface is locked to the direction it arrived, so the scene survives as a coherent, if warped, image.
Curvature is what turns that mirror into a wide-angle one. On a flat mirror you would see only the narrow cone of the scene that happens to reflect toward the camera. On the strongly curved cornea, each patch of surface tilts a little differently, so neighboring patches reflect rays from wildly different directions, and a broad set of scene directions gets compressed onto the small visible cap of the cornea, limited by the viewing geometry and eyelids. The price of that wide field of view is resolution: an entire room's worth of directions is squeezed into a few hundred pixels of iris in even a high-resolution portrait. The eye is, optically, a panoramic camera with very low effective resolution, which is exactly the trade we should expect from a found optical element rather than a designed one.
38.2.2 The geometry: from a corneal pixel to a direction in the world⧉
To invert the reflection we need to know, for each bright corneal pixel, which direction in the surrounding world it came from. That is a chain of geometry, and it starts with a gift the anatomy hands us for free: the boundary between the iris and the white of the eye, the limbus, is very nearly a circle with an approximate physical radius of $5.5\,\text{mm}$, or a diameter of about $11\,\text{mm}$, with variation between people. A known circle in 3-D, viewed obliquely, projects to an ellipse in the image, and the shape and orientation of that ellipse encode how the circle is tilted and how far away it is. Fitting an ellipse to the limbus therefore helps estimate the pose of the eye under a calibrated camera and anatomical model: the position of the corneal sphere's center and the direction the eye is facing (its optical axis, which differs from the visual axis used to define gaze). The rounder the imaged limbus, the more the eye looks toward the camera; the more eccentric the ellipse, the more it is turned away. This is the standard foreshortened-circle cue for tilt, used everywhere a known shape meets perspective.
With the pose in hand, model the cornea as a sphere of known radius centered at that recovered point. Now each ray is determined. A pixel in the eye corresponds to a ray leaving the camera; intersect that ray with the corneal sphere to find the surface point it strikes; at that point the sphere has a known surface normal $\mathbf{n}$ (it points radially outward from the center); and define the unit view vector $\mathbf{v}$ to point from this surface point back toward the camera. With the unit outward normal $\mathbf{n}$, the reflected direction toward the environment is
Sweep that calculation across every corneal pixel and you have mapped the warped reflection onto the sphere of directions around the person: a wide-angle image of the scene they face, unwrapped from the eye. This is the heart of Nishino and Nayar's The World in an Eye Nishino & Nayar 2004: a portrait in, a panorama of the surroundings out (Figure 38.2.1).
The limits are real, in the spirit of L7.8 (Big Lessons). It is one small mirror, so the recovered image is low-resolution and noisy. It is convex and frontal-biased: you only recover directions that reflect toward the camera, a forward-facing cap of the environment, not the full sphere. The sphere model is an approximation because the cornea is really a flattened asphere, and errors in the assumed radius or the fitted pose warp the unwrapping. And the specular reflection sits on top of the iris texture, which contaminates it (more on separating them below). The result is never a good picture of the surroundings; it is a demonstration that the picture is there at all, encoded in the curve of an eye.
38.2.3 What the recovered reflection is good for⧉
The most direct use is the one the geometry promises: recovering the surrounding scene. From a single portrait you can reconstruct a coarse picture of the room the subject is standing in and facing, which is the part of the world behind the camera. It is the eye as accidental panoramic camera, the catadioptric sibling of the Chapter 38.1, Accidental cameras and the Chapter 38.5, Corner camera: another way to see what is not in the frame, this time using a person as the sensor.
The ellipse fit estimates eye orientation. Estimating where the subject is looking also requires the offset between optical and visual axes, ideally calibrated for that person. The reflection can indicate what lies near that direction, but gaze alone does not establish attention. That couples a what-they-see reading to a where-they-look reading from one photograph. And the same facts make corneal imaging a forensic tool: the reflection in a photographed eye can contain the people and place in front of the subject at the moment of the shot (bystanders, a room, a photographer), recoverable from a high-resolution image even when they are nowhere in the visible frame. It is a literal case in which the eye reflection contains scene information outside the frame, with the caveat the whole part insists on: the image is faint, low-resolution, and only as trustworthy as the geometry and the prior that recovered it.
38.2.4 Eyes for relighting⧉
There is a second, quite different way to read the corneal reflection: as a sample of the light in the room rather than a picture of it. Every direction the cornea reflects toward the camera tells you how bright the environment is in that direction, which is precisely what an environment map records. So the same eye that gave us a panorama of the surroundings also gives us a wide-field-of-view measurement of the illumination falling on the face: a light probe, harvested from an ordinary portrait with no chrome sphere held up to the scene. This is Nishino and Nayar's Eyes for Relighting Nishino & Nayar 2004 (Eyes for Relighting). The environment map estimates incident light; relighting the face additionally requires its geometry and reflectance. With a model, you can relight the face under modified illumination, or insert a synthetic object into the photograph and light it consistently with the real scene, so that its shading and shadows match, using illumination estimated from the eye reflection (Figure 38.2.3).
This makes the eye a found light dome. A controlled Chapter 35.5, Light domes surrounds a subject with hundreds of programmable lights to measure a reflectance field, to actively probe how a face responds to light from every direction. The eye does the dual thing passively: it records the illumination that was already present, from every direction it can reflect, in a single frame. Both provide illumination from many directions, but one is active and the other is passive, one engineered and exact, one accidental and approximate, which is the contrast this whole part keeps drawing between built cameras and found ones.
The caveat is the same specular/diffuse problem, and it bites hardest here. The corneal specular reflection, the part that actually images the environment, sits on top of the iris, whose colored, textured diffuse layer adds its own pattern and contaminates the probe. To use the eye as a clean light probe you must first separate the specular reflection from the iris texture beneath it, a specular/diffuse split. One of the most reliable handles on that split is polarization: a specular mirror reflection is partially polarized in a way diffuse body reflection is not, so measurements at several polarizer orientations can help estimate the polarized component, using the separation developed in Chapter 34.12, Polarization imaging. Near normal incidence the reflection is only weakly polarized; polarization alone does not guarantee a clean separation. The probe also has intrinsic limits, including low resolution and frontal bias, since you only ever see the light the cornea throws back toward the camera. The picture is consistent with everything else in this part: a real measurement of the invisible, faint and partial, redeemed by a model and a prior rather than by good optics.