10.7 Locally adaptive regression kernel (LARK)⧉
The bilateral filter Tomasi & Manduchi 1998 settled the central question of edge-preserving smoothing, namely which neighbors count as the same surface, with one device: weight each neighbor by an affinity read off its value difference, so a pixel sitting just across an edge gets almost no say. The lasting idea there is not the weighted average; it is the affinity. The average is merely one thing to do with it, and a single value difference is merely one way to compute it. The locally adaptive regression kernel takes the affinity at its word and promotes it from a number to a shape.
10.7.1 Change the verb: from averaging to fitting⧉
Begin by changing the verb. Instead of telling the operator to average its neighbors, tell it to fit, that is, to regress a smooth function through the nearby pixels and report that function's value at the center. The clean pixel is then what a smooth local model predicts, and the noise is the residue the model declines to explain. This single stance covers two jobs at once. For denoising, evaluate the fitted model at a pixel you already have, and you get a cleaned version of it. For interpolation and upsampling, evaluate the same model at a location between samples, and you get a principled in-between value. Both are the same act: fit locally, then read off the model. That is why kernel regression is a natural common engine for them.
Concretely, around the pixel of interest we suppose the image is locally well-described by a smooth function (a constant, a plane, or a low-order polynomial in the spatial coordinates), and we fit that function to the neighbors by weighted least squares, then take its value at the center. The weights are supplied by a kernel: a neighbor close to the center counts heavily, a far one counts little. With a fixed, round, isotropic kernel this is exactly an ordinary weighted average (for a constant model) or a smooth local polynomial fit (for higher orders), and it is no better than a Gaussian blur. It smears edges just as badly, because a round kernel centered on an edge reaches indiscriminately to both sides and averages the dark surface with the bright one.
10.7.2 Steering the kernel to the local structure⧉
The fix is steering-kernel regression, the locally adaptive regression kernel of Takeda, Farsiu & Milanfar (2007): let the shape of the kernel be dictated by the image. The mechanism is a tidy bit of geometry. At each pixel, estimate the local structure, in practice from the local gradient or, more robustly, from the structure tensor, a local average of gradient outer products. That tensor's eigenvectors point along and across the dominant edge direction, and its eigenvalues say how strongly oriented the neighborhood is: large and lopsided on a crisp edge, small and balanced on a flat patch. From it we read the edge's direction, its strength, and its sharpness.
Now reshape the kernel accordingly. Stretch it along the edge and squeeze it across it. The round weighting becomes an elongated, oriented ellipse that draws many samples down the length of the edge and almost none from the far side. The kernel "steers" itself to the structure: on a flat region, where the tensor is isotropic, it stays round and smooths freely in every direction; along a crisp edge it becomes a thin sliver hugging the contour; at a corner or fine texture, where structure is strong in no single direction, it shrinks so as to trust only the nearest samples. Average along a thin bright filament with such a kernel and the filament survives, sharpened by all the samples taken along its length, while the refusal to reach across it keeps the filament one pixel wide.
This is also why LARK denoises and interpolates so gracefully near structure. When you upsample, the in-between value is predicted from samples that lie along the local edge rather than across it, so a steep boundary is reconstructed crisply instead of being rounded off. That is exactly the failure that betrays naive interpolation.
For a first-order fit about output location $p$, write $x_q=[1,(q-p)_x,(q-p)_y]^\top$. With nonnegative steering weights $w_{pq}$, estimate
A constant fit gives a normalized weighted average. A plane can reproduce a local ramp when the sample geometry has sufficient rank. Use a stable least-squares solve or stated regularization if the neighborhood is poorly constrained.
One generic oriented Gaussian uses a positive-definite matrix $C_p$:
A larger eigenvalue penalizes displacement more strongly in its direction, narrowing support there. An edge-normal direction therefore needs stronger penalty than the tangent. This describes the geometry; the particular LARK estimator specifies how gradient statistics set its eigenvalues. Noise and crossings make that estimation difficult.
10.7.3 The regression flavor of the affinity⧉
Read all of this as the regression flavor of the affinity, the same L10.1 idea in a different form. The bilateral wrote affinity as a scalar weight on a value difference: a single number per neighbor, large when two pixels are alike, small across an edge. LARK writes affinity as the geometry of an anisotropic kernel derived from local structure: not one number but an oriented shape that says, all at once, which directions to trust and how far. The instinct is identical: let the edges decide who gets averaged with whom. But here it is promoted from a single number to an oriented shape, and that shape explicitly describes orientation and scale. The bilateral knows only how different a neighbor is; the steering kernel also knows in which direction the surface continues, so it can keep averaging confidently along an edge that the bilateral would only know to stop at.
The relationship is easy to overstate, so be precise about it. The bilateral's range weight and LARK's steering kernel are not the same object computed two ways; they are two different ways to spend the same affinity instinct. The bilateral compares the center pixel's value to each neighbor's value directly. LARK never compares the two values at all; it looks only at the local structure (the gradients) and builds a shape from it, then fits. That is why LARK belongs to the same family as the bilateral while sitting a clear step away from it: same question, different mechanism, more expressive answer. For the wider family (patches, optimization energies, guided fits), see the methods that follow in this part; here the point is only that regression is one more place the affinity can live.
10.7.4 LARK as a structure descriptor⧉
There is a second, almost incidental use that falls out of the same object, and it deserves a sentence because it shows how general the construction is. The steering kernel computed at a pixel is, in effect, a compact descriptor of the local image structure. It encodes the orientation, strength, and isotropy of the neighborhood in a single small matrix. Collected over a window and normalized, these locally adaptive kernels form a feature, the LARK descriptors of Seo and Milanfar, used for detection and matching: finding a template object in a cluttered image, or spotting an unusual action in video, by comparing these structure descriptors rather than raw pixels. We only note it. The same machinery that steers a denoising kernel can serve as a similarity feature: once you have a faithful description of local structure, you can either filter with it or recognize with it.