9.2 The expert's eye as evaluation🔗⧉

You have two versions of a denoiser. Before recruiting viewers, look at both on a set chosen in advance. Check whether one has erased hair, shifted color, or left structured noise in shadows. These observations may settle an implementation decision. If the tradeoff is subtler, they help you formulate the question for a study.

9.2.1 Judge it yourself first🔗⧉

A study of “which looks better” can mix several judgments: sharpness, color, naturalness, or resemblance to an intended appearance. Participants may also use different displays and viewing conditions. A number collected under those conditions needs to be interpreted accordingly.

For development, you can control the display, crop, zoom, and comparison order yourself. This makes inspection cheap and repeatable. It does not make one developer a representative population. Use it to find problems and understand tradeoffs before deciding what evidence a broader claim requires. Chapter 9.4, Towards meaningful user studies explains how to make that decision.

Your own inspection is especially valuable when you can name the defect and show where it occurs. Recruiting more viewers before identifying that question can produce an expensive answer about sharpness when the decision concerned ringing or color. First use the control and experience you already have to make the alternatives precise. Then ask what remains unknown about the audience.

9.2.2 You are the expert, in a narrow sense🔗⧉

Implementing a demosaicker gives you practice finding zipper artifacts and false color. Use that knowledge. Mark the affected edge, show the crop, and explain the defect so another person can check it.

Expert inspection has a long history in vision, including the observations behind Mach bands and opponent color mechanisms discussed in Chapter 3.4, Spatial and temporal vision and Chapter 4.5, Advanced color: opponents, perceptually-uniform, and preferred. But detecting a defect and measuring its importance to other viewers are different questions. Your familiarity can help you find a faint halo while making you unusually sensitive to it. Expectations about your own method can also bias the comparison.

For example, after studying ringing you may recognize a faint bright-dark pair beside a roofline that an untrained viewer cannot name. Mark the location, show the reference and residual, and explain how the filter could produce it. That is a testable defect claim. It does not establish that viewers prefer the alternative: another version may remove the ring while blurring roof tiles. Expertise helps formulate the comparison and identify failure mechanisms; preference still depends on the viewer, task, and competing defects.

9.2.3 Training your eye🔗⧉

Learn the artifacts and their causes: zipper patterns in demosaicking, ringing after sharpening, halos around strong edges, banding in gradients, and loss of texture after denoising. A name helps you describe a problem precisely; a crop and a controlled comparison make the observation checkable.

Align the images and toggle them in the same location. This makes small changes easier to locate than repeatedly shifting your gaze between panels. Also inspect the images separately: flicker can make an otherwise unobtrusive difference conspicuous, and a visible difference need not be a defect. An amplified difference image helps locate changes but does not measure their visibility at normal contrast.

Compare at one image pixel per display pixel, then at the intended output size and viewing distance. Keep the display transform and crop fixed. Hide method labels and vary presentation order where possible. Record observations before seeing the scores in Chapter 9.3, Image metrics. Take breaks and return to the comparison with fresh eyes.

A practical comparison uses two views of the same evidence. Toggle aligned images in the same place to locate changes, then inspect each version without flicker at the intended viewing size. The toggle makes a small displacement or halo easy to find; the separate view tells you whether it is objectionable in use. An amplified difference image supplies a third view that can expose a faint color bias across a smooth region. Its amplification is a diagnostic choice, not the contrast at which a viewer sees the defect.

9.2.4 The input set decides what you can see🔗⧉

Choose inputs for three purposes.

• Trivial inputs check specific expectations: a constant, impulse, edge, or ramp. As in Chapter 9.1, Developing, testing and debugging, passing these tests establishes those cases, not general correctness.
• Representative photographs cover the ordinary scenes and capture conditions the method is meant to serve. Include multiple cameras and a range of subjects, illumination, and skin tones where relevant.
• Difficult cases probe predicted failures: fences and fabric for demosaicking, backlit faces for tone mapping, dark moving subjects for denoising, and highly saturated colors for gamut handling.

Keep a fixed regression set and reserve an untouched evaluation set. Repeatedly tuning on a test set turns it into development data, however diverse it was initially.

Check the intended display conditions too. Ambient light, screen luminance, display processing, and output scale all affect visibility. Change them deliberately and record them, rather than attributing every difference to the algorithm. An OLED panel or a bright room does not make every artifact more visible; some differences become harder to see.

9.2.5 Beware the average, look at the worst case🔗⧉

A mean PSNR can improve while a demosaicker produces severe false color on one fence. An automatic tone adjustment can improve most photographs while clipping a backlit face. Averages answer a useful question about the chosen distribution, but hide which images fail and how badly.

Inspect poor results as well as reporting the mean. Include percentiles or the fraction of images with a specified failure, and show representative failures alongside typical successes. A metric can help select suspicious cases, but also inspect a sample it ranks well: its blind spots may hide defects there.

The worst five images in one dataset do not establish the worst possible behavior or certify that a product is ready. Track whether failures cluster by scene, camera, or subject, and expand the set when a new failure appears. Apply the same selection rule to competing methods so that the comparison does not become a collection of favorable examples.

9.2.6 Knowing when to stop, and when to ask🔗⧉

Repeated inspection can make a familiar defect easy to overlook: anyone who has shipped an image pipeline has seen a colleague glance at the screen and name, in two seconds, the artifact that has been there for three weeks. It can also make you care disproportionately about a tiny difference you have spent weeks optimizing: spend a fortnight on chroma noise and you will find it in every photograph, including the ones that are fine. Compare against saved baselines, take breaks, and ask someone unfamiliar with the method to describe what they see before revealing your concern.

Escalate to a study when the decision depends on other people's preferences, ability to detect a difference, or performance on a task. If two results seem indistinguishable, first ask whether distinguishing them matters enough to justify measurement. A simpler or faster method may already be the appropriate choice.

Choose evidence to match the claim. Pixel error measures agreement with a specified reference; expert inspection identifies artifacts; a suitably designed study measures responses from a defined audience under stated conditions. They complement one another, and none automatically overrides the others.