18.11 Evaluating tone mapping🔗⧉

Two tone-mapped versions of one radiance map, side by side. Which is better?

The answer depends on the objective: preferred appearance, similarity to a viewed scene, preservation of structure, or performance on a visual task. There is no single universal target image, but each stated objective supports useful comparisons.

18.11.1 Why the comparisons disagree🔗⧉

Twenty years of comparison studies have not produced an agreed ranking of tone-mapping operators, and the reason is less the difficulty of running experiments than the question they ask. Kuang and colleagues compared eight operators by paired comparison and found Durand and Dorsey's bilateral operator and Reinhard's photographic operator best overall Kuang et al. 2004. Ledda and colleagues compared six operators against the same scenes shown on a high-dynamic-range display, and declined to rank them at all Ledda et al. 2005. Yoshida and colleagues had observers compare operators with the real scenes, viewed in place Yoshida et al. 2005. Čadík and colleagues rated operators on separate attributes, brightness, contrast, detail, color and artifacts, and related those to overall quality Čadík et al. 2008. These studies asked about preference, about fidelity to a displayed reference, about fidelity to the real scene, and about attributes; there is no reason their answers should agree. "Which operator is best" is underdetermined until we say which of the goals of Part 18, Tone mapping and display we mean: fidelity to the scene's appearance, maximum visibility of detail, or preference. And most of these studies ran each operator at its default parameters, while a single operator's own parameter can move its result further than the differences between operators (Figure 18.11.1⛶ fullscreen).

There is no ground truth for the third goal, and only an expensive one for the second: an appearance match needs the scene, or a display that reproduces it, beside the result. Objective metrics fill part of the gap. HDR-VDP models early vision, from the optics of the eye through contrast sensitivity at several scales, and predicts whether a difference between a reference and a test image is visible and how much it degrades quality Mantiuk et al. (HDR-VDP). It needs absolute, calibrated luminances and a description of the display. It tells whether a difference can be seen, not whether it is better: it cannot express preference, it does not model the appearance effects of Chapter 18.1, Tone mapping and perception beyond contrast visibility, and a tone-mapped image, which is supposed to differ visibly from its HDR input, is not what it was built to judge. Metrics designed for tone mapping, such as TMQI below, combine structure with a statistical model of naturalness for that reason.

18.11.2 What can be measured without a reference🔗⧉

Some things are facts about the output and can be checked.

Structural fidelity compares the rendered output with the HDR input under a specified model. TMQI combines a multiscale structural measure with a statistical naturalness measure (Yeganeh & Wang 2013 (TMQI)). Its structural component uses a reference; it should not be grouped with purely no-reference tests or treated as a direct measurement of all retained information.

TMQI's naturalness component models output brightness and contrast statistics. It supplies a limited statistical preference model, not a general detector of overprocessing or a universal aesthetic score.

Halos, local contrast reversals, banding and clipping can be inspected directly. Gradient-sign changes can help locate suspicious regions, but noise, alignment and intentional local edits complicate that diagnostic. Confirm flagged regions visually rather than treating every reversal as an equally severe error.

Consistency. We can run the operator on two frames of the same scene that differ slightly and see whether the result changes a lot. An operator with strong local adaptation can be unstable in exactly this way, which matters for video and shows up as flicker, the same failure Chapter 13.10, Video denoising describes.

18.11.3 What needs people🔗⧉

Everything else needs viewers, and the tone-mapping literature has a long history of doing it properly, which is worth borrowing from.

Paired comparison is the workhorse: show two results of the same scene and ask which is preferred. It is easier for a viewer than a rating, it produces a ranking through the usual scaling models, and it needs a lot of comparisons.

An appearance-matching study can compare the rendering with a controlled HDR display or a viewed scene. Such a study has to state adaptation, luminance, surround and viewing geometry. This asks a different question from simple preference.

Task-based evaluation asks whether viewers can read a sign, count objects or distinguish a material. It measures accuracy and response time under representative conditions. Preference can be reported separately when it also matters.

Two practical warnings. Preferences depend heavily on the display and the viewing environment, so a study run on one screen in one room transfers less than people assume. And preferences differ by content: the operator that wins on architectural interiors is not the one that wins on faces, which is why an average over a mixed test set can rank an operator that is nobody's favorite above one that is somebody's.

18.11.4 What to report🔗⧉

For a tone-mapping result, the minimum useful report is: the input radiance map and its dynamic range; the operator and every parameter, since these operators have many and their values are most of the result; the display the result was made for, including its peak luminance; a structural check against the input; and at least one artifact check, with haloes and gradient reversals named specifically.

If a claim of superiority is being made, it needs people, on a stated display, with a stated task or preference question, and with the content range described, since a single scene proves nothing about an operator.

And a small discipline that costs nothing: publish the radiance maps. A tone-mapping result cannot be checked, reproduced or re-rendered without the input, which is why the field's shared scenes have been worth more than most of its operators.