42.3 Lightroom-style raw developers🔗⧉

Drag the exposure slider in a raw developer and a 50-megapixel photograph brightens under your hand, twenty times a second. The editor did not rewrite the raw file, which it never modifies, and it did not process 50 million pixels twenty times a second either. What it did is the subject of this chapter: a raw developer is a recipe store wrapped around a pipeline that can be evaluated at many resolutions, and nearly every architectural decision in one follows from making that evaluation fast, faithful, and reproducible.

Lightroom and Capture One are the commercial examples; darktable darktable, RawTherapee RawTherapee and its derivative ART ART are open source and can be read. We use a third, CompPhotoRoom, the browser raw developer built for this book and embedded in Chapter 8.13, A raw developer in the browser. Its design notes record why each decision was taken and what it measured, which is what makes it useful here.

42.3.1 The recipe and the catalog🔗⧉

Chapter 8.12, Recap ISP, non-destructive editing set out the model: the original samples are never modified, and an edit is a list of parameters for a fixed pipeline, recomputed on demand (Big lesson L8.6show). A system built on that model has two stores.

The recipe belongs to one photograph: white balance, exposure, curve points, mask geometry, the crop. It is small, a few kilobytes. Editors write it into a sidecar file beside the raw, usually XMP, an XML metadata format that Lightroom uses under Adobe's camera raw namespace and darktable uses with its own history stack; RawTherapee writes its own profile files. A sidecar makes the edit portable: copy the folder and the edits travel with it.

The catalog belongs to the collection: which files exist and where, ratings, flags, keywords, collections, and a copy of each recipe, indexed so that "all five-star photographs from 2024 with a person in them" is a query rather than a scan of a hundred thousand sidecars. Lightroom Classic and darktable both keep their catalog in an SQLite database. The two stores can disagree, and every editor has rules for which one wins when a sidecar changes behind the catalog's back.

CompPhotoRoom runs in a browser, which changes where these stores can live. Originals stay where they are on disk and are reached through the File System Access API, whose handles persist between sessions; nothing is copied or uploaded. Derived data (proxies, thumbnails, cached intermediates) goes into the browser's private file system, and the catalog into IndexedDB. Recipes and ratings can be written out as XMP sidecars that Lightroom reads.

That last step shows how much a recipe depends on its interpreter. Lightroom read CompPhotoRoom's first sidecars for their keywords and ignored every slider. The missing field was a flag telling Lightroom whether the numbers were still to be applied or already baked into the pixels; with no flag it assumed baked. And once Lightroom did apply them, the same numbers meant different things: CompPhotoRoom's contrast of 20 corresponds to about 38 in Lightroom, measured by developing test images in both and fitting one to the other. A recipe is a program for a particular pipeline. Moving it to another pipeline is a translation, and the translation has to be measured, not assumed.

42.3.2 The pipeline and its order🔗⧉

Under the recipe sits a fixed sequence of stages: read and linearize the raw, subtract black level, white balance, demosaick (Chapter 15.1, Basic traditional demosaicking), denoise, convert camera color to a working space, tone and color adjustments, local adjustments under masks, sharpening, geometric corrections, and an output transform to the display or file (Chapter 8.12, Recap ISP, non-destructive editing). The order is part of the algorithm. White balance before demosaicking changes the demosaicker's color-difference estimates; sharpening before a tone curve sharpens different values than sharpening after it.

Editors differ in how much of this order they expose. Lightroom shows panels in a fixed order and hides the actual pipeline order behind them. darktable shows its modules as a list that is the pipeline, lets advanced users reorder it, and since its 3.x releases defaults to a scene-referred workflow: values stay proportional to scene light through most of the pipeline, and the compression to display range happens once, near the end, rather than being applied early and undone later (Chapter 8.3, Tone point operations).

42.3.3 Case study: CompPhotoRoom🔗⧉

Three requirements shaped CompPhotoRoom, and they pull against each other.

1/ Real raw files, edited non-destructively.

2/ Large images. A 60-megapixel RGB image in 32-bit float is 722 MB, and the full set of local Laplacian coefficients for it about 2.5 GB, against a 4 GB address space for WebAssembly in the browser. Full-resolution RGB can therefore never exist during editing.

3/ Swappable algorithms: several demosaickers, several denoisers, each selectable, without giving up the speed of compiled code.

The design answers each with one idea.

One graph, three drivers🔗⧉

The stages form one graph, and three drivers evaluate it.

The stage code is the same in all three; only the region and the resolution change. Lightroom Classic has the same structure under other names: baked previews for the library grid, an editable Smart Preview of about 2560 pixels for developing, and a cache of post-demosaic, pre-adjustment data. CompPhotoRoom's cache boundaries sit in the same places.

A proxy is only useful if it predicts the export, so every stage declares how it behaves across resolutions, its scale contract:

• invariant: a point operation, same parameters at every scale;
• radius: a fixed-radius filter whose radius shrinks with the proxy factor;
• pyramid: parameters attached to absolute full-resolution pyramid levels, so a proxy at one quarter scale runs levels 2 and up and skips levels 0 and 1. The proxy shows exactly the part of the effect visible at that zoom and converges to the export as you zoom in;
• native: not representable at proxy scale at all. Denoising is the example, since the noise it removes does not survive downsampling; the interface says so and asks for a 1:1 view.

Lightroom's advice for the last class is the same: zoom to 1:1 to judge noise reduction and sharpening. The scale contract makes the rule explicit per stage, and for pyramid stages removes the need for it.

A raw mosaic adds a twist. Opening a 50-megapixel Bayer file on a quad average (one RGB pixel per 2x2 block) is instant, but that proxy has a different demosaic than the export. CompPhotoRoom opens on the quad average, then rebuilds the proxy in the background from the same Malvar–He–Cutler demosaic the export uses Malvar et al. 2004, in bands of about 2 megapixels, yielding between bands so sliders keep responding, and swaps it in when done. The screen, the 1:1 view and the exported file then come from one demosaic rather than three.

Boundaries at algorithms, not operators🔗⧉

Each stage boundary costs an intermediate image written to memory and read back. At proxy resolution about six boundaries cost about 25 ms of memory traffic; twenty stages at full resolution would cost half a second. So boundaries go where a reader would say "a different method": demosaic, denoise, sharpen, local contrast, tone mapping. Each is its own stage and its own compiled entry point, swappable at run time.

Everything that is merely a knob collapses into one fused point stage: exposure, contrast, curves, the hue/saturation/luminance bands, color grading, vibrance, vignetting, grain. The fused stage has a fixed shape (per-channel tables, a color matrix, more tables, the profile's hue and saturation lattice, the grading lookups), so it can be compiled ahead of time even though its contents change with every slider. The interpreter composes each run of per-channel adjustments into a table once per parameter change, and a slider at its neutral value becomes an identity table, costing one lookup and no branch. On a 0.89-megapixel proxy this took the point stages from 82.1 ms to 4.0 ms.

Geometric operations get the same treatment in their own domain. Crop, straighten, perspective, rotation, lens distortion and lateral chromatic aberration compose into one map from output to source coordinates, applied in a single resampling pass (Chapter 11.1, Warping and resampling). Applied one after another they would interpolate the photograph four times, and each interpolation loses sharpness that no later stage can restore.

Spatial stages, the local Laplacian filter (Chapter 10.4, Local Laplacian filters) and detail processing, cannot be fused with their neighbors, because each output pixel needs a neighborhood of inputs. They are where swapping algorithms is interesting, and where compiled code matters most.

Two implementations of every stage🔗⧉

Every stage exists twice: a readable JavaScript reference and a Halide kernel compiled to WebAssembly (Chapter 29.3, Halide at a high level: decoupling algorithms from schedules). The JavaScript chain is checked bit for bit against a Python reference, and the Halide kernels are checked against the JavaScript within 8-bit quantization, so the whole chain is anchored: Python, JavaScript, Halide, WebAssembly. The reference also does useful work in the product. It runs the application before a kernel exists, and it is what the "?" panel on each slider shows the reader.

The checks catch what no single implementation reveals. A test that ran each kernel on very small images found that four of five read outside their buffers below 8 pixels on a side. The same class of bug had earlier caused a hang. With threads enabled, the second Highlights adjustment of a session sometimes never returned. Every hypothesis concerned the large full-screen render. The cause was a thumbnail: the local Laplacian recursed until an image side was a few pixels, the kernel trapped on a 2x3 input, and in the threaded build the trap escaped one worker thread while the others waited forever for it. The fix was one line refusing inputs under 8 pixels. The lesson for systems is to test the degenerate sizes a pipeline will inevitably produce, thumbnails and pyramid tops, not only the sizes a person looks at.

Merging is most of the performance🔗⧉

It is tempting to credit CompPhotoRoom's speed to compiled kernels. The measurements say otherwise: most of the gain came from merging, the fused point stage and the single resample, which cut memory traffic. The kernels matter in the spatial stages, and there the schedule matters more than the language; threading the local Laplacian gave 1.8x rather than the four to eight hoped, because its schedule materialized every pyramid level and pushed tens of megabytes of intermediates through memory (Chapter 29.2, Modern CPUs: memory hierarchy, parallelism, and what it takes to go fast).

Export follows the same rule at a larger scale. A frame too large to develop whole is developed in horizontal bands with overlapping margins for the spatial stages; a 50-megapixel raw exports in a few seconds in two bands, and the memory ceiling follows the device, so a small machine refuses with an explanation rather than crashing.

42.3.4 What a raw developer knows about cameras🔗⧉

A raw developer carries per-camera data the user never sees. A camera profile is several things: a color matrix from camera to a standard space, a hue and saturation correction lattice, a look table, and a tone curve. Profiles are camera specific; near neighbors are not interchangeable, and CompPhotoRoom's first comparison against Lightroom flattered itself by using one camera's profile on its successor's files. Lens profiles add distortion, vignetting and chromatic aberration models per lens. Adobe's DNG format carries much of this inside the file (Chapter 45.14, DNG: the Digital Negative).

This data is also what makes two developers look different on the same raw. Matched against Lightroom on two test photographs, CompPhotoRoom's default rendering first differed by a mean ΔE of 8.5 and 4.7 (Chapter 9.3, Image metrics). Reading the camera's own matrix and correction tables from its DNG profile brought those to 5.9 and 2.9 without touching any algorithm. Much of the visible difference between two raw developers is camera data and default settings.