8.12 Recap ISP, non-destructive editing🔗⧉

An image signal processor turns sensor measurements into a finished rendering. Its stages interact: black offsets affect white balance, interpolation changes noise statistics, and a tone curve changes the appearance of sharpening. The useful pipeline description names not only the operation, but also its input domain and assumptions.

8.12.1 A basic ISP🔗⧉

An ISP can combine dedicated hardware, software, and learned components. A pipeline diagram describes dependencies, not one mandatory ordering shared by every camera. Figure 8.12.1⛶ fullscreen gives a representative route from a CFA raw image to a display file.

Decode the raw and its metadata. Identify the active area, CFA phase, sample packing, linearization, black and saturation levels, and color calibration. A raw file may already contain corrections or reconstructed channels. Chapter 8.11, File formats and compression describes the storage choices.

A raw decoder supplies measurements and metadata; a developer also chooses reconstruction and appearance. Keep those roles distinct when comparing library outputs. A convenience “postprocess” call may include white balance, color conversion, scaling, and encoding.

One representative pipeline is:

Capture control and sensor correction. AE, AWB, and autofocus commonly use processed preview statistics in a feedback loop, rather than being completed before any processing. Linearize stored samples when required, subtract the calibrated black level, and correct known defects. For sample $s$ with black level $b$ and saturation code $w$, a common normalization is $x=(s-b)/(w-b)$, with $w>b$. Preserve useful negative black-relative fluctuations and keep saturation information separately.

Reconstruct color. Part 15, Demosaicking and joint methods estimates missing channels from the CFA. White-balance gains can be applied to the mosaic before this step, and noise reduction can precede reconstruction or be combined with it. These choices must agree with the reconstruction model.

Reduce noise. Use parameters appropriate to the current domain. The simple raw variance model does not fully describe noise after interpolation, data-dependent merging, or nonlinear rendering. Part 13, Denoising explains these changes. A final display-domain cleanup can also be useful if it models the data it receives.

Correct the lens response. Lens shading is a spatial gain, often corrected in the mosaic domain; it also amplifies noise. Lateral chromatic-aberration correction and distortion correction resample coordinates and can be combined with reconstruction or later geometric processing. Their placement is an implementation choice with consequences for interpolation and noise.

Set white balance and convert color. Apply channel gains in their calibrated linear basis and map camera channels into a working color space. A matrix is an approximation to colorimetric conversion, not a guarantee for every reflectance and illuminant. Unequal gains and a general color matrix do not commute. Chapter 4.2, White balance and white point develops the calibration and adaptation assumptions.

Render the appearance. Choose exposure, contrast, highlight roll-off, and color treatment. Controls can operate in scene-linear, log, perceptual, or display-referred spaces, provided the conversions are explicit. Rendering parameters can be learned from examples Bychkovsky et al. 2011. Noise reduction and sharpening may share a detail decomposition; judge the combination at the final viewing scale.

Prepare the output. Convert to the destination gamut and output encoding, with a defined policy for out-of-range values, then quantize and compress. Tone mapping has already changed scene relationships; applying an output transfer function is a distinct step. A nonlinear tone curve does not leave the image scene-linear merely because the final sRGB encoding has not yet run.

Figure 8.12.2⛶ fullscreen illustrates intermediate results on one capture. State the conversion used to display each intermediate array. A green cast in reconstructed, unbalanced camera RGB is caused by channel sensitivities and illumination, not simply by the number of green sites. Demosaicking has already filled all three channels at every output location.

8.12.2 Two stages this part did not teach🔗⧉

Two blocks in the diagram above have no chapter in this part, and their absence is deliberate rather than an oversight. Demosaicking and denoising are not optional garnish on an ISP: a Bayer sensor measures one color per photosite, so without demosaicking there is no color image at all, and at the light levels a phone works in, a frame without noise reduction is unusable. They are also the two stages where the most engineering effort goes, and where the pipeline's ordering question is sharpest.

The reason to hold them back is that neither fits in a pipeline tour. Each carries its own body of methods, its own failure modes and its own literature, and each gets a part.

For reconstruction, Part 15, Demosaicking and joint methods starts with Chapter 15.1, Basic traditional demosaicking, which builds the edge-directed green estimate and the color-difference trick that every classical demosaicker refines, then covers Chapter 15.5, Alternative color filter arrays for quad-Bayer and X-Trans, Chapter 15.6, Learned demosaicking, and Chapter 15.3, Mosaics and aliasing for why the mosaic aliases in the first place.

For noise, Part 13, Denoising begins with Chapter 13.1, Noise models, transforms, and benchmarks, which is where the affine noise model of the sensor chapter becomes something you can calibrate and test against. Chapter 13.2, Denoising by averaging covers the burst case, Chapter 13.3, Gaussian, median, and bilateral filtering the single-image filters, Chapter 13.4, Pyramid and wavelet denoising coefficient shrinkage, and Chapter 13.7, Learned denoisers what replaced all of it in production.

The two stages also interact, which is the deeper reason not to teach either as one box in a chain. Demosaicking correlates the noise it inherits, and denoising before reconstruction destroys the very gradients the demosaicker uses to pick its interpolation direction. Running them in either order is a compromise, and Chapter 15.8, Joint demosaicking and denoising treats them as one estimation problem instead. A production ISP usually does the same.

8.12.3 Real ISPs🔗⧉

Real implementations use different domain boundaries and scheduling. A raw editor may keep a scene-linear working representation while individual modules convert temporarily to log or perceptual coordinates. A display-oriented filter may run after the tone transform. Read the module contract and pipeline version before treating a product name as a complete description.

Useful comparison questions are where sensor corrections occur, what the reconstruction expects, which color basis each gain or matrix uses, and what representation enters denoising and sharpening. Those questions transfer across hardware ISPs and desktop developers.

The main domain choices can be summarized without prescribing one product-specific order:

• Scene-linear representations retain light-proportional values for exposure, color conversion, and compositing.
• Log or perceptual representations support relative-contrast and appearance controls with explicitly defined conversions.
• Display-referred representations support output-oriented sharpening, color cleanup, and compression.

The principle is to match arithmetic to its purpose. Combining physical light requires linear-light values. Appearance controls may deliberately use another domain. A scene-linear editor can still contain nonlinear tools and a display transform; “linear workflow” does not mean every intermediate remains proportional to scene radiance.

The open-source implementations surveyed for this book expose several concrete architectures. The table describes the inspected pipelines, not an invariant order for every version or processing mode.

The table explains why “denoise before gamma” is an incomplete description of a real system. Raw denoising can exploit a calibrated sensor-noise model. Later luma/chroma denoising works on reconstructed and rendered values, often to suppress objectionable color variation before delivery. They address different representations and can coexist. Likewise, a raw editor’s scene-referred architecture does not prevent a particular module from using log or Lab internally.

The smaller reference designs are especially useful to read. Mini-ISP exposes the distinction between color conversion and output transfer encoding in a short chain. Infinite-ISP shows what a more extensive chain does after that transition. ART’s documented per-module spaces make deliberate conversions visible. An application assembled from generic array operations requires the same scrutiny: a variable named RGB or BGR says nothing about whether its values are linear, encoded, or already tone mapped. The color-side account is developed in Color TLDR#Color and encoding in the real world.

8.12.4 Pipeline design and tuning🔗⧉

A real ISP also needs joint tuning. Sam Hasinoff emphasized the following complications in discussing this material. Several stages can affect the same visible property, making isolated slider tests insufficient:

• The blocks are not independent. A linear chain of stages is nice conceptually, but in practice the effects of the blocks are deeply entangled: any interesting property of the final image (local contrast, say, or how skin reads, or the color of a sunset) is usually the net result of several blocks interacting, not the output of one. Change the tone curve and you change what the sharpener bites on; change the white balance and you change what the saturation stage amplifies. That entanglement is what makes tuning so hard: there is rarely one knob for one visible effect.
• Which space should a block operate in? Many operations are defensible in more than one domain: linear scene light, a tone-mapped space, the final display-referred values, or all of the above. Sharpening in linear light, in gamma, or after tone mapping gives three different looks; the "right" choice is a design decision, not a given, and different cameras answer it differently.
• Should a block anticipate the future, or report its past? A stage can be tuned to anticipate downstream processing (e.g. demosaic gently because a strong sharpener follows), or it can stay neutral and carry forward rich metadata about what it did, so later stages can compensate. Both are real architectures with real trade-offs.
• Can tuning be canonicalised across cameras? Every sensor and lens is a little different, so a tuning that sings on one body may not transfer to another. Whether you can build one canonical tuning that works across a product line (or must hand-tune each) is an open, expensive question.
• Or throw most of it away? The most radical option is to replace much of this hand-tuned chain with end-to-end learning: train a single model from raw to finished image and let it discover the entangled interactions the block diagram hides. Which brings us to how ISPs are changing.

8.12.5 The ISP, evolving: traditional → learned → generative🔗⧉

Hand-designed, learned, and generative components can coexist in one pipeline. They offer different control and validation requirements:

• Hand-designed stages expose interpretable parameters, but interactions still require tuning.
• Learned stages estimate operations from training data, individually or jointly. This does not imply that a named phone pipeline is one end-to-end network.
• Generative stages can synthesize plausible detail under uncertainty. Their appearance may improve while fidelity to the particular captured scene decreases; see Chapter 40.1, Human factors and the art of photography.

For any learned stage, test camera calibration, out-of-distribution scenes, temporal behavior, and detail fidelity. A visually convincing output is not evidence that every feature was measured.

Replacing one stage with a network preserves the surrounding interfaces: a learned demosaicker can still emit camera RGB for a separately calibrated color transform. Replacing a larger chain lets the training loss coordinate its interactions, but moves control into the training setup and model. A request such as “reduce magenta in these shadows” may then require a conditioning control, new examples, or retraining instead of one exposed matrix or curve. This is a change in how the system is designed and debugged, not merely a change in the formula inside a box.

8.12.6 Recap 2: non-destructive editing optional🔗⧉

Non-destructive editing stores changes separately from the original samples. It works with raw or rendered inputs, although a rendered input has already committed to earlier processing decisions.

The stored recipe may live in a catalog, a sidecar, or metadata alongside the image. It describes operations, parameters, and masks. The editor recomputes affected stages as needed and caches intermediate results and previews. The original pixel data can remain unchanged even when metadata in the same file changes.

The same raw can then yield a web JPEG and a print by rerunning the pipeline at each output's size, sharpening and color space, and a decision made in the first month can be revised in the sixth. The price is recomputation every time the image is shown, which is why these editors cache previews at several sizes and rebuild them when the recipe changes (Chapter 42.3, Lightroom-style raw developers). The pattern reaches beyond photography: a video editor's edit decision list records cuts and effects over untouched source footage and renders them on demand, and any workflow that would rather replay a derivation than store its output makes the same trade.

Because the recipe is independent of the raw, the same file can become entirely different finished pictures: you simply feed the pipeline a different parameter set, and nothing about the original is consumed in the process (Figure 8.12.3⛶ fullscreen).

Changing parameters avoids repeated lossy saves of intermediate images. Reproduction also requires the original, profiles, external masks, model weights, and the processing version. A future software release may interpret the same sliders differently. Store those dependencies, and keep an exported reference when exact appearance matters.

The demo below lets you run the develop engine directly. It is a working miniature of exactly this pipeline (every stage of the chapter, wired to a Lightroom-style panel) with the edit recipe printed live beside it (Figure 8.12.4⛶ fullscreen).

A non-destructive editor exposes pipeline decisions as an editable recipe. Its screen preview may use reduced resolution or approximations; inspect a full-resolution export for scale-dependent denoising, sharpening, masks, and color conversion. Chapter 8.13, A raw developer in the browser provides a practical setting for those experiments.

The benefits follow directly. The parameter edit is reversible: drag exposure back to zero and it is as though you never touched it, because the original was never altered. It is re-editable: you can revisit a three-month-old adjustment and change a single slider with no accumulated loss from repeated re-saving. The recipe can drive a thumbnail preview or a full-resolution export. Spatial radii and resolution-dependent stages must be scaled appropriately, and the preview can be an approximation to the final rendering. The one thing it demands is speed: recomputing the pipeline every time you nudge a slider only feels interactive if the pipeline is fast, which is why these tools lean hard on the graphics processing unit (GPU) and on carefully optimized image code, a concern we return to in the performance part.