BookNotesSlidesP-sets Syllabusdraft · v0.2.49.1
◀▲⌂▶
💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.

3.7 Human vision TLDR⧉

Every photograph in this book ends at a viewer, and the viewer is not a light meter. The eye is a camera with an uneven sensor and a very narrow window of sharp vision; behind it sits a system that throws away most of the light's absolute level, recodes color into a brightness channel and two color-difference channels, reports surfaces rather than illumination, and fills in what it did not look at. Almost every engineering choice downstream, from white balance and gamma to chroma subsampling, tone mapping, frame rates and image metrics, is a bet on one of those behaviors. This chapter lists them, one paragraph each, with the chapter that develops it.

3.7.1 The eye is a camera with an uneven sensor⧉

The optics are a camera's: the cornea does most of the focusing, the lens adjusts it for distance, and the pupil sets the aperture. The sensor is not. Cones, which work in daylight and see color, are packed tightly in the fovea, a small central pit where acuity is highest; rods, which work in dim light and do not see color, fill the periphery and are absent from the fovea itself. Sharp, colored vision therefore covers only the few degrees you are looking at, and everything else is coarse. Where the optic nerve leaves the retina there are no receptors at all, which leaves a blind spot about fifteen degrees toward the temporal side of each eye's field; you do not see a hole because the other eye covers it and the brain fills in the rest. Details: Chapter 3.1, The eye and image formation.

fig-eye-cross-section
Figure 3.1.1 (reproduced). The eye as a camera. Cornea and lens form the image, the pupil sets the aperture, and the retina is the sensor, with the fovea at the center of gaze and the optic disc, where the nerve leaves, off to one side.
fig-cone-rod-distribution
Figure 3.1.2 (reproduced). Where the receptors are. Cones peak sharply at the fovea and thin out quickly; rods dominate the periphery and vanish at the fovea; the optic disc has neither.

3.7.2 Three cones reduce a spectrum to three numbers⧉

Color vision starts with three kinds of cone, called L, M and S for long, medium and short wavelengths. Each cone's response is the light's spectrum weighted by that cone's sensitivity curve and summed:

$$ L = \int C(\lambda)\,\bar l(\lambda)\,d\lambda, $$

and likewise for $M$ and $S$. In words, a cone response is a dot product between the spectrum reaching the eye and the cone's sensitivity. Everything the spectrum carried beyond those three projections is lost, so many different spectra produce the same triple and look identical. They are called metamers, and metamerism is what lets a display with three primaries reproduce a color without reproducing its spectrum. It is also why two fabrics that match in the shop can mismatch in daylight: each match is a match between reflectance times illuminant, and changing the illuminant changes the product. Details: Chapter 3.2, Perceptual color and trichromatic vision.

fig-cone-sensitivities
Figure 3.2.1 (reproduced). The three cone sensitivities. The L, M and S curves overlap heavily, L and M most of all, so each cone responds to a broad band rather than to one wavelength.
fig-metamers
Figure 3.2.9 (reproduced). Metamers. Two physically different spectra whose cone responses are equal, and which therefore look the same.

3.7.3 Primaries are a choice, and the axes are skewed⧉

Because the cone curves overlap, no light excites one cone alone: the three axes are not orthogonal, and since light cannot be negative, only part of the cone space is reachable. Red, green and blue are therefore not the colors of nature but a convenient engineering choice of three lights to add, just as cyan, magenta and yellow are a convenient choice of three inks to subtract. Matching a color with three chosen primaries sometimes needs a negative amount of one of them, which is the geometric fact that makes every real display's gamut smaller than what the eye can see. Details: Chapter 3.2, Perceptual color and trichromatic vision.

3.7.4 Color blindness is a missing or shifted axis⧉

When one cone type is missing, the space loses a dimension and colors that differ only along that axis collapse into one; when a cone's sensitivity is shifted, the axis shrinks rather than vanishing. Red-green deficiency, from the L or M cone, is by far the most common form, affecting roughly one man in twelve and one woman in two hundred among people of Northern European descent (it is carried on the X chromosome, hence the imbalance). For anyone designing an image, the consequence is practical: never let red against green be the only difference that carries meaning. Details: Chapter 3.2, Perceptual color and trichromatic vision.

fig-ishihara
Figure 3.2.14 (reproduced). An Ishihara plate. A numeral drawn in dots that differ from the background mainly along the red-green axis, invisible to a reader missing that axis.

3.7.5 After the cones, brightness and two color differences⧉

The cone signals are not sent to the brain as they are. They are recombined into one channel that sums them, roughly brightness, and two that take differences, roughly red against green and blue against yellow. That opponent recoding is why no color looks reddish-green, and it is the model behind the engineering spaces that separate luminance from chrominance, YCbCr in video and JPEG, Lab for color differences. The model is a useful approximation rather than a set of exact perceptual hue axes. Details: Chapter 3.2, Perceptual color and trichromatic vision.

fig-opponent-channels
Figure 3.2.17 (reproduced). Opponent channels. The three cone signals recombined into a summing channel and two difference channels, the structure that luminance-chrominance spaces imitate.

3.7.6 The eye works on ratios, not levels⧉

Across adaptation the eye covers about fourteen orders of magnitude, from starlight to snow in sunlight, but at any one instant it sees only a small slice of that range, set by where it has adapted. Within that slice, what it detects is relative: the smallest visible increment is a roughly constant fraction of the background (Weber's law), $\Delta Y / Y \approx k$ for luminance $Y$. A gray step that is obvious in the shadows is invisible in the highlights at the same absolute size. Scaling all the light by a common factor therefore leaves most of what we see unchanged, and vision is largely invariant to multiplication. That single fact is why images are stored with gamma or log encodings that spend their code values on ratios, and why photographic exposure is counted in stops. Details: Chapter 3.3, Adaptation and constancy.

fig-photopic-scotopic-range
Figure 3.3.4 (reproduced). The range across adaptation. One log-luminance axis from starlight to sunlight, with the regimes where rods, both, and cones do the work.
fig-weber-fraction
Figure 3.3.6 (reproduced). Weber's law, measured on yourself. Lower an increment until it disappears against its background; the fraction you land on stays roughly the same as the background changes.

3.7.7 The eye reports surfaces, not light⧉

What the visual system is after is the surface: its reflectance and color, not the light that happens to fall on it. It discounts the illumination, both its level and its color, closely enough that a white shirt stays white from daylight to tungsten, and it uses the surroundings to decide what is lit and what is painted. The checker-shadow illusion makes two identical grays look different because one is read as a light square in shadow, and the same machinery makes a gray look lighter on black than on white. Constancy is an inference from context, not a readout, and when the cues are ambiguous two people can infer different lights, which is what happened with the dress. For photography the consequence is that white balance is a rendering decision for the viewing conditions, not a measurement to be undone. Details: Chapter 3.3, Adaptation and constancy.

fig-checker-shadow
Figure 3.3.9 (reproduced). Adelson's checker-shadow illusion. Squares A and B have the same value; the one read as lying in shadow looks lighter, because the eye reports the surface rather than the light.
fig-the-dress
Figure 3.3.17 (reproduced). The dress. An ambiguous photograph whose colors are read differently by viewers who assume different illuminants.

3.7.8 Spatial vision is band-pass, and measured in degrees⧉

The eye is not equally sensitive to all spatial frequencies. Its contrast sensitivity peaks at a few cycles per degree and falls away on both sides: very fine detail is lost to optics and receptor spacing, and very slow gradients are discounted by the same processing that makes vision work on ratios. Because it is defined per degree of visual angle, not per pixel, viewing distance decides what is visible: step back from a print and fine texture disappears first. Details: Chapter 3.4, Spatial and temporal vision.

fig-campbell-robson
Figure 3.4.2 (reproduced). The Campbell-Robson chart. Frequency rises to the right and contrast downward; the boundary where the stripes vanish traces your own contrast sensitivity, and it moves when you change your viewing distance.

3.7.9 Color detail is coarser than brightness detail⧉

For the two color-difference channels, sensitivity is low-pass and cuts off at a much lower frequency than for brightness, so fine color variation is far less visible than fine luminance variation. Blur the color channels of a photograph hard, keep its brightness, and most viewers cannot tell. JPEG, video and every camera pipeline spend that gap through chroma subsampling. It holds for typical photographs rather than as a law: colored text, isoluminant edges and saturated fine patterns are where the loss shows. Details: Chapter 3.4, Spatial and temporal vision.

fig-blur-chroma-luma
Figure 3.4.5 (reproduced). The gap, done to a photograph. The same picture with its color channels blurred and its brightness kept, beside the reverse; only the second looks damaged.

3.7.10 Vision has a finite temporal bandwidth⧉

Flicker above a certain rate fuses into steady light, which is why a display can refresh and a lamp can modulate without our noticing. That rate depends on brightness, the size of the flickering area and where it falls on the retina, which is why a flicker you cannot see straight on can catch the corner of your eye. There is no universal 60 Hz limit, and flicker fusion does not by itself set the frame rate that makes motion look smooth: exposure time, the number of distinct images, the display refresh and the light's modulation are four different things. Details: Chapter 3.4, Spatial and temporal vision.

3.7.11 Vision is foveal, serial, and partly invented⧉

Because only the fovea sees sharply, the eye moves several times a second, placing the fovea on what the task needs and building the scene from a sequence of glimpses. The periphery guides where to look next. What we believe we saw is a construction assembled from those glimpses, context and expectation: large changes to a picture can go unnoticed if they happen during a blink or away from where the viewer looked (change blindness), and the blind spot is filled in rather than reported. For images this cuts both ways. A viewer will not inspect most of a photograph closely, which is what saliency and composition exploit; and an artifact in the right place is noticed immediately. Details: Chapter 3.5, The visual system attends little and makes things up.

fig-scanpath
Figure 3.5.1 (reproduced). A scanpath. The fixations of one viewer on one picture, clustered on faces and text, with fast jumps in between.

3.7.12 What the rest of the book does with this⧉

Each of these behaviors is spent somewhere in the pipeline, and knowing which one explains a design that would otherwise look arbitrary.

Other animals solve the same problem with other optics, other receptors and other priorities, which is a useful reminder that none of these behaviors is the only possible design. Details: Chapter 3.6, Animal eyes.