15.8 Joint demosaicking and denoising🔗⧉

A noisy raw mosaic contains missing color components and uncertain measured samples. Demosaicking and denoising can therefore benefit from a shared estimator. A denoiser that ignores the CFA mixes unlike colors; a demosaicker that ignores noise can make poor direction decisions and spread their errors.

These are failures of particular models and interfaces, not proofs that either sequential order must fail. CFA-aware denoising can preserve color phases, and an RGB denoiser can model correlations introduced by demosaicking. Joint estimation retains access to the original observations and avoids treating an intermediate estimate as exact. Its benefit must still be measured against suitable sequential baselines.

For a fixed linear demosaicker $A$ and raw noise covariance $\Sigma$, the output covariance is $A\Sigma A^\mathsf{T}$. Reused samples correlate output values. Simple averaging of independent equal-variance samples reduces variance, but negative correction weights or noisy adaptive choices need not. Thus post-demosaic noise is not universally quieter, and a raw-domain scalar variance estimate does not fully describe it.

Follow a possible error on a striped shirt through a sequential pipeline. A demosaicker produces a patch of false color where its local model cannot resolve the texture. A subsequent denoiser may preserve that patch because it is spatially coherent, even while removing independent grain around it. Sharpening can then increase its contrast. Each operation can be reasonable for its assumed input while the composite result reinforces an earlier mistake. This is a reason to retain the original mosaic and its uncertainty when designing the estimator. It is not inevitable: a sequential method whose later stages model reconstruction artifacts can behave differently.

The information-flow issue is the commitment at an intermediate image. Once a stage passes only one RGB estimate onward, a later stage may have no indication that two different textures fitted the original mosaic almost equally well. It cannot reconsider that ambiguity using the raw likelihood unless the raw data or an adequate uncertainty representation remains available. A joint objective keeps those constraints active while the estimate changes. This is the useful distinction between coordinating several reconstruction operations and merely running them once in sequence.

15.8.1 The two orderings, made visible🔗⧉

Compare the same raw input, output domain and computation budget. Tune the sequential and joint methods separately on validation data.

Denoising before reconstruction can remove details needed for interpolation if it oversmooths them. A CFA-aware denoiser need not mix neighboring color channels, and its strength can be chosen to retain those structures.

Demosaicking first changes the noise covariance and may introduce structured artifacts. A denoiser trained or designed for this input can address some of them. Independent white-noise assumptions should not be transferred unchanged from raw samples to demosaicked RGB.

15.8.2 The joint formulation🔗⧉

Model the measurement as a noisy mosaic,

Here $M$ selects the measured component from a clean camera-color image. The variance is conditional on $x$, with independently distributed errors as an optional approximation. Raw noise can also have calibrated correlations and defects.

The recovery is the usual data-fit plus prior,

With fixed positive covariance $\Sigma$, this is a weighted least-squares data term. If the Gaussian approximation instead uses $v_i(x)=a(Mx)_i+b$, the full negative log likelihood includes the changing normalization:

Both terms depend on $x$. Freezing variances from a pilot estimate gives a useful weighted least-squares approximation, but omitting the log term while optimizing signal-dependent variances is not the same likelihood. Require $v_i>0$ on valid samples. At low photon counts or clipping, use an appropriate Poisson/read-noise or censored model (Chapter 2.8, Noise, signal-to-noise ratio and dynamic range).

The prior can couple spatial structure and color channels. Learned and explicit priors express different image assumptions; neither automatically fits every scene. A learned prior can also be told the noise level at each pixel as an extra input map, as FFDNet does for denoising Zhang, Zuo & Zhang 2018 (FFDNet); the sketch below does that with the calibrated raw noise model.

# joint demosaick + denoise, the shape of the learned version
inp   = pack_mosaic(y)                    # 2x2 tile -> 4 planes at half resolution
pilot = nonnegative_signal_estimate(y)
level = pack_mosaic(sqrt(a*pilot + b))                # standard deviation, same packed layout and units
xhat  = net(concat(inp, level))           # joint reconstruction in the chosen output domain
# the noise map is the FFDNet idea from the denoising part: the network is
# told how noisy each pixel is instead of having to infer it

15.8.3 FlexISP: one energy, one prior🔗⧉

The idea of collapsing the pipeline into a single optimization was crystallized by Heide et al. (2014) in a system named FlexISP, a flexible image-signal processor built as one energy function rather than a fixed chain Heide et al. 2014. Its architecture is exactly the inverse-problem template we have been using. On one side, a single image-formation model: a linear operator $A$ that composes all the physical steps the sensor performed (subsampling by the color-filter mosaic, blurring by the optics' point-spread function, geometric warps for lens distortion and lateral color, even the resampling of a burst of frames) into one map from the clean latent image $x$ to the raw measurement $y$. On the other side, regularization applied to the shared latent image, combining image and cross-channel structure. Recovery is the one objective $\tfrac12\lVert Ax-y\rVert^2 + \lambda\,\Phi(x)$, minimized as one large optimization.

A composite operator permits the data fit to account for several physical effects together. Operator splitting decomposes the numerical solve into manageable subproblems; it does not remove the need to define each forward operation, boundary and adjoint consistently. Chapter 13.11, Denoising as a universal prior distinguishes explicit proximal priors from arbitrary inserted denoisers.

A shared final-image prior avoids some incompatible intermediate objectives, but its accuracy remains a modeling choice. FlexISP demonstrates a flexible formulation, not a theorem that one prior always beats every stagewise pipeline.

FlexISP evaluated ADMM Boyd et al. 2011 (ADMM) and a first-order primal-dual solver, and used the latter in its reported examples. Its regularization combines BM3D Dabov et al. 2007, total variation and a cross-channel term rather than relying on a color penalty alone (FlexISP paper). Splitting the numerical work into updates does not recreate a one-pass ISP: the updates revisit one shared image and reconcile their constraints through iteration. This is why a system can contain several computational modules while still solving a joint reconstruction problem.

15.8.4 What it buys, and where it is deployed🔗⧉

Gharbi and colleagues evaluate learned joint reconstruction against classical methods, with particular attention to difficult patches (Gharbi et al. 2016). Mobile-imaging challenges include related tasks such as remosaicking and denoising (MIPI 2022 (quad-Bayer remosaic challenge) ⚠️); identify each track's data construction and noise protocol before calling it a real-camera test.

Published systems establish that joint raw reconstruction is practical. They do not reveal the architecture or processing path of every flagship phone, which may change with capture mode and software version.

Choose the input convention with the model: CFA layout, black-level subtraction, gains, clipping and noise units must match training or calibration. White balance can precede or be included in reconstruction if its effect on the noise is modeled. Camera metadata supplies useful parameters, but calibration uncertainty and the unknown signal can still require estimation.

15.8.5 Correcting chromatic aberration jointly🔗⧉

Lateral chromatic aberration provides a concrete interaction: color channels can depict the same scene edge at different sensor positions, weakening a demosaicker's shared-edge assumption.

Start with the optics. A real lens does not bend all wavelengths equally: its glass is dispersive, so red, green, and blue focus and, more relevantly here, magnify slightly differently (the full catalog is in Chapter 22.1, Aberrations and optical challenges, and the cures in Chapter 22.2, Aberrations correction). Lateral (or transverse) chromatic aberration is the magnification version: because the three colors are imaged at very slightly different scales, a scene point away from the optical center lands at a slightly different radius in each channel. At the image center the three coincide; toward the corners they separate, radially. At a high-contrast edge near the frame edge (a dark window frame against bright sky, exactly the worst case) the red edge, the green edge, and the blue edge can land several pixels apart (Figure 15.8.3⛶ fullscreen). In a simplified example each channel is sharp but displaced. Real lenses can combine lateral displacement with defocus and other aberrations.

Color-difference reconstruction uses the fact that channels often share edges. When those edges are displaced, their differences contain narrow bands of contrast. Interpolating the differences can spread or distort these bands (Figure 15.8.3⛶ fullscreen). Independent-channel interpolation and explicitly aberration-aware methods make different assumptions.

Rescaling demosaicked channels can correct much of a lateral displacement, but may leave artifacts introduced during reconstruction. Including per-channel warps in the forward model lets the estimator use the raw samples at their calibrated positions. This motivates joint correction; it does not prove that every sequential correction is too late or that only a joint solver can work.

The general construction composes per-channel warps with optical blur and CFA sampling.

Biscarrat et al. Biscarrat et al. 2026 build a learned version of this idea. Once lateral chromatic aberration is described as a per-channel warp, the red and blue samples of a raw image no longer sit on the pixel grid, and their offsets grow toward the corners. Rather than resampling them onto a grid first, their network keeps each sample at its calibrated sub-pixel position and predicts, for every output pixel, a continuous reconstruction kernel evaluated at the nearby samples. The kernel adapts to local image content and noise level. The same architecture merges an aligned raw burst, where sub-pixel offsets come from hand motion instead of the lens, and it demosaicks, denoises and corrects lateral chromatic aberration from a single raw frame in one step.

Imagine a high-contrast vertical edge whose red image falls slightly to the right of its green image. A demosaicker that assumes co-located edges may interpret the displaced red samples as a genuine color change and interpolate accordingly. A later channel warp can align the main edges while leaving some of that interpolation error. Including the displacement in the sampling model lets the reconstruction compare samples at their appropriate locations from the start.

This motivates a joint treatment, but does not make every sequential correction invalid. A raw-domain correction or a demosaicker designed to tolerate displaced channels can also work. The concrete advantage of the joint model is that its data term knows the geometry while estimating the missing colors.

A displaced step makes the color-difference problem explicit. Let green rise from dark to bright at one column, and let red make the same transition two columns later. Between those positions, green is bright while red is still dark, so $R-G$ contains a narrow negative band even though the surfaces on either side are neutral. Interpolating this band under a smooth-color-difference assumption can spread it to neighboring pixels. Bringing the reconstructed channels into alignment afterward corrects their displacement but need not undo that interpolation error. Including the calibrated channel positions during reconstruction lets the estimator compare samples that refer to the same scene location.

15.8.6 The argument generalizes🔗⧉

Raw burst merging, demosaicking and deblurring share related measurement constraints. A joint estimator can use them together; a staged design can also pass calibrated uncertainty and validity information forward. Rendering remains a separate choice even when reconstruction is jointly optimized (Chapter 15.9, Joint methods and the end-to-end pipeline).