BookNotesSlidesP-sets Syllabusdraft · v0.2.49.1
◀▲⌂▶
💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.

17.7 Encoding HDR images⧉

A merged radiance map is a floating-point measurement. Its values are proportional to scene radiance, they routinely exceed one, and their range can reach twenty stops or more. An ordinary 8-bit file holds none of that: it clips everything above white, and its gamma-encoded codes run out of precision long before the shadows of such a scene. Storing HDR images therefore needs encodings of its own, and there are two different jobs. The first is to keep the measurement, scene-referred and relative, for later processing, compositing or tone mapping. The second is to deliver a finished photograph to displays whose range differs from one viewer to the next, while still showing something sensible on a standard screen. This chapter covers the formats for both. How the second kind of file is rendered on a particular display is the subject of Chapter 18.10, HDR displays and display mapping.

17.7.1 How many bits does a range need?⧉

The eye notices a change in luminance of roughly one percent over most of its range (Chapter 3.3, Adaptation and constancy), so a good encoding keeps the step between neighboring codes below about one percent of the value it encodes. A linear integer encoding has the same absolute step everywhere, which means the relative step is tiny in the highlights and coarse in the shadows. To keep one-percent steps down to the darkest value of a range of $N$ stops, the step must be one hundredth of that darkest value, and the brightest value is $2^N$ times larger, so a linear code needs about $100 \times 2^N$ levels: 21 bits for 14 stops, 27 bits for 20.

A logarithmic encoding spends its codes evenly in ratio instead. Steps of one percent give $\ln 2/\ln 1.01 \approx 70$ codes per stop, so 20 stops need about 1,400 codes, 11 bits. This is the argument of Chapter 4.3, Encoding: linear, gamma or log, where the gamma of standard images is a cheap approximation of the same idea over a range of a few hundred to one. Every format in this chapter is a way of getting log-like precision over a wide range: floating point, which is logarithmic in its exponent and linear in its mantissa; a shared exponent; an explicit logarithm of luminance; or a transfer function shaped by measured contrast sensitivity.

Figure 17.7.1 puts every format of this chapter on that scale.

fig-hdr-encoding-precision
Figure 17.7.1. The relative step between neighboring codes, against luminance over about 43 stops, for each encoding in this chapter, every curve computed code by code from the format's definition and ending where its range ends. The gray band is about 1%, roughly where a step becomes visible. 8-bit sRGB and 16-bit linear integers (white at 100 cd/m²) are fine only near white and grow coarse toward black. Half float keeps a step of $2^{-10}$ to $2^{-11}$ of the value, a sawtooth, over its normal range from $6.1 \times 10^{-5}$ to 65,504, about 30 stops, with coarser subnormal values below (dashed). Radiance RGBE's brightest channel steps by 0.4 to 0.8% over about 76 orders of magnitude, but a channel at a tenth of the brightest, as in a saturated color, steps by 4 to 8%. LogLuv 32 is flat at 0.27% over 38 orders and LogLuv 24 at 1.1% over 4.8 orders. PQ, absolute from 0.0001 to 10,000 cd/m², is near 1% at 10 bits above about 10 cd/m² and much coarser in the deep shadows, where the eye's own threshold rises; 12 bits are about four times finer. Scene-referred formats are placed with 1.0 at 1 cd/m².

17.7.2 Floating point: half floats and OpenEXR⧉

A floating-point number stores a sign, an exponent and a mantissa, so its relative precision is the same at every magnitude, set by the mantissa, and its range is set by the exponent. The 16-bit half float has one sign bit, five exponent bits and ten mantissa bits. Its relative step is $2^{-10}$, about 0.1%, well under the visibility threshold, and its normal numbers run from about $6.1 \times 10^{-5}$ to 65,504, some 30 stops; subnormal numbers extend the range down to $2^{-24}$ with decreasing precision. Three half floats per pixel, 48 bits, hold any radiance map a camera can produce with room to spare, and negative values, which appear after some color conversions, are representable.

OpenEXR, released by Industrial Light & Magic in 2003, is the standard container for such data in film production and in computational photography Kainz, Bogart & Hess 2004 (OpenEXR). A file holds any number of named channels, each in half, 32-bit float or 32-bit unsigned integer, so depth, alpha, normals or per-pixel weights can travel with the color. Lossless compression schemes (ZIP, and PIZ, a wavelet coder that does well on noisy photographs) typically halve the size; lossy ones (B44, DWAA and DWAB) trade a controlled loss for much smaller files. Images can be tiled and stored at several resolutions, a file can carry several parts such as the views of a stereo pair, and deep images store several samples per pixel. The academy color encoding system (ACES) uses OpenEXR with its own wide primaries for interchange between studios. What OpenEXR does not record by default is what the numbers mean: a merged bracket should carry its primaries and white point, which the header supports, and whether its scale is relative or calibrated in absolute units (Chapter 17.2, Camera response and radiance calibration).

17.7.3 Shared exponents and logarithms: Radiance RGBE and LogLuv⧉

Before floating-point image files were common, Ward's Radiance renderer stored HDR images in 32 bits per pixel by giving the three channels one shared exponent: an 8-bit mantissa each for red, green and blue, and an 8-bit exponent common to all three, the RGBE format Ward 1991 (RGBE). The exponent gives a range of about 76 orders of magnitude, far more than any scene, and the mantissas give a precision of about 1% relative to the largest channel. The weakness is the other two channels: in a saturated color, the small channels share the large one's exponent and keep only a few bits of precision, and the format cannot store negative values. The .hdr files of Radiance, still the usual format for environment maps used to light rendered scenes, are RGBE with run-length compression; an XYZE variant stores CIE XYZ instead of RGB.

LogLuv separates luminance from color and encodes each the way vision needs Ward Larson 1998 (LogLuv). Its 32-bit form stores a sign and a 15-bit logarithm of luminance, in steps of 0.27% over 38 orders of magnitude, and the CIE $u'v'$ chromaticity coordinates in 8 bits each, a uniform enough chromaticity space that 8 bits per coordinate keep color steps near the threshold of visibility. Measured as errors in RGB, the chroma quantization moves the smaller channels by 1 to 3%, more than RGBE does for most colors, though RGBE's worst case, a deep saturated blue, loses far more. Its 24-bit form has a 10-bit log luminance, steps of 1.1% over 4.8 orders of magnitude, and a 14-bit index into a table of chromaticities. Both live inside TIFF. LogLuv is perceptually the most efficient of the classic formats, but floating point won on convenience, since processing code works directly on its values.

Figure 17.7.2 draws the bits of one pixel in each layout.

fig-hdr-bit-layouts
Figure 17.7.2. One pixel's bits in four formats, to scale. Half float RGB spends 48 bits, a sign, a 5-bit exponent and a 10-bit mantissa per channel, for about 30 stops at a step under 0.1%. Radiance RGBE spends 32, three 8-bit mantissas and an exponent they share, which buys about 76 orders of magnitude at the price of coarse steps in every channel but the brightest. LogLuv 32 spends its 32 on a sign, a 15-bit logarithm of luminance and 8 bits each for the chromaticity $u'$ and $v'$, a 0.27% step over 38 orders. A 10-bit PQ pixel spends 30 bits on display luminance from 0.0001 to 10,000 cd/m², with steps near 1% at the bright end.

17.7.4 Editable HDR: floating-point DNG and TIFF⧉

A photographer who merges a bracket usually wants to keep editing it as if it were a raw file: set white balance, exposure and the tone curve later, non-destructively (Chapter 42.3, Lightroom-style raw developers). Version 1.4 of the DNG specification, in 2012, added floating-point samples in 16, 24 and 32 bits for this purpose (DNG specification). A merged DNG holds linear, demosaicked, scene-referred values together with the camera's color metadata, so a raw developer applies the same color processing as for a single exposure and the photographer edits the merged picture with the usual controls. Floating-point TIFF, with 32-bit samples, serves the same role in tools that do not read DNG, without the camera metadata.

17.7.5 Backward-compatible HDR: from JPEG-HDR to gain maps⧉

A floating-point file is useless to a viewer that cannot read it, and for years that was nearly every viewer. Ward and Simmons's JPEG-HDR stored an ordinary tone-mapped JPEG, which any viewer could show, and hid in the file's application markers a low-resolution ratio image between the radiance map and that rendition, so an HDR-aware decoder could multiply its way back to the full range Ward & Simmons 2004 (JPEG-HDR). JPEG XT later standardized backward-compatible HDR extensions of JPEG along the same lines Artusi et al. 2016 (JPEG XT). The gain map is the current form of the idea, with one change of purpose: what it reconstructs is not the scene's radiance but a second rendition, prepared for a display with more headroom.

For photographs, gain maps provide a base rendition plus data for adapting its range. Both the base image and the HDR rendition need testing in the actual delivery path.

A gain-map file stores a base rendition, an auxiliary image and reconstruction metadata. An SDR base provides a usable fallback in decoders that ignore the auxiliary data. The gain map describes a relationship between renditions, not an absolute display luminance by itself.

The map may have one or several channels and may be lower resolution than the base. Its bounds, offsets, gamma and display-capacity metadata are part of the reconstruction. Full gain depends on available HDR headroom relative to the reference white and the encoded capacity range, not on one universal peak-nit threshold. Android's Ultra HDR documentation specifies its conventions and discusses ISO 21496-1 interoperability (Ultra HDR format).

For a simplified positive, offset-free example in matching linear-light color coordinates, define

$$G=\log_2(H/S),\qquad R_w=S\,2^{wG},\quad 0\le w\le1.$$

Here $S$ is the SDR base, $H$ the target HDR rendition, and $w$ the display adaptation weight. Zero weight returns the base; full weight returns the target. This is geometric interpolation, not an arithmetic blend of encoded pixel values. Real formats include offsets and metadata scaling, so their specified equations apply, not this simplified expression on JPEG code values (Adobe gain-map documentation).

We compute the relationship before reducing the map's resolution and inspect edges where the SDR and HDR renderings differ sharply. A gain map need not be low frequency: downsampling can produce halos or color fringes. A constant map must be handled without dividing by a zero min/max range.

We validate the final file on an HDR path and on an SDR fallback path, and check that resizing, sharing and re-encoding preserve the intended base and auxiliary metadata. A screenshot or an SDR simulation cannot demonstrate the physical luminance of an HDR display.

Backward compatibility is a useful way to understand the file design. A viewer that reads only the SDR base can display a deliberately prepared SDR rendition. A viewer that understands the auxiliary map and metadata can use them to produce a different rendition appropriate to its available headroom. The map stores a relationship to the base instead of another independent full-resolution RGB picture. In a simplified ratio-only example, a gain of three means that the corresponding linear-light value in the other rendition is three times the base value; it does not alone mean a particular number of nits. The reconstruction rules and display mapping are still needed to determine emitted light.

The encoding recipe is useful even when a library handles the file format. We put the two renditions into matching linear-light coordinates and choose offsets that keep both numerator and denominator positive. We compute the relationship at their common resolution:

$$G=\log_2\frac{H+k_H}{S+k_S},\qquad u=\frac{G-G_{\min}}{G_{\max}-G_{\min}}.$$

For an illustrative convention with encoding exponent $\gamma>0$, store $q=\operatorname{round}(255u^\gamma)$. The bounds, offsets and exponent are recorded in the metadata. At full reconstruction strength, decode

$$\widehat G=G_{\min}+(G_{\max}-G_{\min})(q/255)^{1/\gamma}, \qquad \widehat H=(S+k_S)2^{\widehat G}-k_H.$$

Quantization and compression make this approximate. A constant map needs a separate branch because its bounds coincide. We reduce the map's resolution only after computing the ratio, and check the resulting reconstruction around edges. This example fixes an exponent convention for teaching; a production encoder must follow its format's gamma, offsets and display-adaptation rules.

The residual representation is itself a research choice. Canham and colleagues investigate gamma maps, which encode an exponent relating the SDR and HDR renditions, rather than a multiplicative gain (Gamma Maps). Gain-MLP instead fits a small network per image to represent the map, replacing ordinary bitmap compression with an implicit representation. Its purpose is to reduce reconstruction artifacts and storage, not to infer new scene measurements (Gain-MLP project). These are two separate choices: what relationship to encode, and how to store it.

Figure 17.7.3 takes one radiance map apart as a gain-map file.

fig-gain-map-anatomy
Figure 17.7.3. A radiance map of an office with a bright window, stored as a gain-map file. The SDR base is the photograph tone-mapped into the standard range. The gain map, stored at a quarter of the resolution, holds the log₂ ratio of an HDR rendition to the base, from 0 indoors to 2 stops on the window, the content boost. A display with headroom $H$ applies the map with weight $\log_2 H / 2$: an SDR display (1×) shows the base, a 2× display half the gain, and a 4× display all of it. The renditions are shown divided by their headroom so that they fit this page: as the headroom grows, the window keeps its brightness while the room falls relative to it. At the window's edge, a map stored at 1/16 of the resolution spreads the window's gain onto the frame beside it, a halo; at full resolution the edge stays clean. The base and the HDR rendition are built here from the radiance map; the gain is applied on luminance.

17.7.6 Display-referred encodings: PQ, HLG and video⧉

HDR video and most HDR still formats other than gain maps carry display-referred values: the code says how bright the pixel should be on the display, not how bright the scene was. Two transfer functions, both specified in ITU-R BT.2100, replace the standard gamma.

The perceptual quantizer, PQ (SMPTE ST 2084), encodes absolute luminance from zero to 10,000 cd/m². Its curve was fitted to Barten's model of contrast sensitivity Barten 1999 so that, with 12 bits, every step between codes stays below the threshold of visibility at every luminance Miller, Nezamabadi & Daly 2013 (PQ). With 10 bits the steps are near 1% only above about 10 cd/m², about 1.1% near 100, and grow to several percent in the deepest shadows, where the eye's threshold also grows. Hybrid log-gamma, HLG, from the BBC and NHK, is relative instead: a square-root curve in the lower half of the signal, close to a conventional gamma, and a logarithmic one above it for the highlights, with the display's own peak deciding how bright the top of the signal is rendered Borer & Cotton 2015 (HLG). HLG offers useful compatibility properties, but it is not ordinary SDR gamma and cannot guarantee a correct result through every SDR chain. Both require proper color and transfer-function interpretation (ITU-R BT.2100).

HDR video delivery uses different structures. HDR10 uses PQ with static metadata; Dolby Vision profiles use additional metadata and profile-specific signaling. These are not interchangeable with a still-image gain map. Editing and export must preserve the transfer function, primaries and metadata. The same transfer functions now serve still images in HEIF, AVIF and JPEG XL, alongside or instead of a gain map. Scene-referred log curves, the S-Log, Log C and V-Log encodings of cinema and mirrorless cameras, are a third family: they record the scene, not the display, in 10 or 12 bits, and a color grade renders them later, as a raw developer would (Chapter 4.3, Encoding: linear, gamma or log).

Figure 17.7.4 stores one radiance map in every format of this chapter and decodes it back.

fig-hdr-format-roundtrip
Figure 17.7.4. One radiance map stored in eight formats and decoded from the stored integers: 8-bit sRGB clipped at an SDR white, 16-bit linear integers, half floats, Radiance RGBE, LogLuv24 and LogLuv32, 10-bit PQ (BT.2100 narrow range on BT.2020 primaries) with a chosen scene value shown at a chosen peak, and a gain-map file whose base is the 8-bit sRGB rendition and whose three-channel map, in 8 bits with gamma 1, has a quarter of its width and height. The map is normalized so its log-average luminance is 0.18; 16-bit linear and LogLuv24, whose range has a fixed top, are scaled so the brightest pixel takes the top code. Left, the decoded image at the chosen exposure; right, the relative error of each pixel on a log scale, in the worst of the three channels or in luminance, with 1% marked. The table gives each format's bits per pixel, its decoded value and error at the probe, and its median and 99th-percentile error over the photograph. Click or drag to move the probe; click a row to show that format. Under the photograph, ramps of deep blue and deep red climb 12 stops to the map's brightest value: RGBE keeps the large channel to 0.4%, and the small channels, which share its exponent, keep two to four bits. LogLuv32 holds luminance to about 0.1% everywhere, while its 8-bit chromaticity moves the small channel of a saturated color by a few percent. Turn the exposure down to see the windows, where the clipped base, the gain map's reduced resolution and the PQ peak each show; turn it up for the shadows, where 8-bit sRGB and 16-bit linear run out of codes. LogLuv24's 4.8 orders of magnitude cannot hold the cathedral nave. JPEG's lossy compression of the gain-map file is not modeled, and bits per pixel are counted before any compression. Radiance maps: Paul Debevec, Byong Mok Oh.

17.7.7 Choosing an encoding⧉

The choice follows from the job. To keep a merged radiance map for later processing, OpenEXR with half-float channels is the default: linear, scene-referred, with its primaries recorded and a note on whether its scale is relative or absolute. To keep editing a merged photograph as a raw file, a floating-point DNG preserves the camera metadata a raw developer needs. Environment maps and older tools still use Radiance RGBE. To deliver a photograph that must look right on standard screens and use the headroom of HDR ones, a gain-map file, JPEG or HEIF, carries both renditions. Video and display-referred stills use PQ or HLG with BT.2020 primaries and their metadata. Across all of them, the encoding has to say what its numbers mean, scene or display, relative or absolute, and in which primaries, or the next program will guess.