10.10 Recap: which edge-aware technique when?⧉
An edge-aware method needs a definition of which structures matter. Intensity differences, patch similarity, local color models and source disagreement answer different questions. None is a general detector of semantic boundaries.
10.10.1 The three relationships to edges⧉
Begin with the three families themselves, each summarized by what it does to an edge and the job it is for.
Gradient-domain editing fits prescribed gradients under boundary or fidelity constraints. Use Chapter 10.8, Poisson image editing to reduce a pasted boundary, accepting that source colors and low frequencies may change.
Edge-aware filters control which samples influence an output. Choose Chapter 10.1, Bilateral filtering for a simple local baseline, Chapter 10.6, Guided image filtering for a local affine guide, and Chapter 10.4, Local Laplacian filters for multiscale contrast control. Chapter 10.2, Non-local means compares patches. Modern denoising choices are discussed in Part 13, Denoising.
Chapter 10.9, Seam optimization chooses paths or labels. Selection, compositing and carving need different energies: following an object edge, switching where sources agree, and removing low-importance content are distinct goals.
The three families treat an edge in three different ways. Gradient-domain methods keep the edges and let the absolute levels float: they reconstruct an image from its differences, so a mismatch at a boundary is spread smoothly over the region. Filters average within regions and stop at edges, so the edge survives because nothing is averaged across it. Discrete methods decide where the edges of the result will be, choosing a path or a labeling. The first two are often confused because both leave edges intact; the difference is that a filter is local and runs in one pass, while a gradient-domain solve is global and every pixel of the region can move in response to its boundary.
Affinity links the second and third families. In a filter, a high affinity between two neighbors means "average these together"; on a graph edge, the same high affinity means "cutting here is expensive", so a cut prefers to pass between dissimilar pixels. The link is conceptual. Colorization, matting, guided filtering and graph cuts each define their weights from a different local model, and their matrices are not interchangeable.
10.10.2 Pros, cons, and cost: when to reach for each⧉
The relationship picks the family; within a job the trade-offs pick the specific tool. The recurring axes are the ones the part kept returning to: local versus global, filter versus optimization, and speed versus quality. Here is each family priced fairly.
Poisson and gradient-domain methods are global and seamless, and sensitive to the encoding they run in. The space you integrate in matters, which is why exposure-mismatched edits and gradient-domain HDR compression work in log rather than linear values (Chapter 10.8, Poisson image editing; Pérez et al. 2003). The cost is a constrained sparse solve: conjugate gradient, multigrid, or a fast Fourier transform on a suitable domain, with the cheaper membrane shortcut (a Laplace solve on the boundary mismatch) available when the guidance field is the gradient of a single source, as in a plain clone. Choose the processing domain and the boundary model explicitly, and use screening to keep a reference appearance. Reach for this family when the artifact lives at a boundary. Its failure mode is the other side of L10.4: because it discards the absolute level, it re-lights and bleeds color when the boundary varies strongly, and it turns a flat region into a smooth membrane. When both halves of a composite are real photographs whose low frequencies should both contribute, as in stitching two exposures of one scene, Laplacian-pyramid blending, which feathers the coarse levels instead of discarding the level, is often the sounder choice. Compare both when exposure or texture differs.
Bilateral filtering and its relatives are mostly single passes, the family's main virtue, but their cost spans a wide range (Chapter 10.1, Bilateral filtering, Chapter 10.6, Guided image filtering, Chapter 10.4, Local Laplacian filters). The brute-force bilateral is slow; the bilateral grid Chen et al. 2007 approximates range-dependent averaging in real time; the guided filter He et al. 2010 evaluates its local affine model in linear time using box sums, though guide mismatch and numerical precision still matter; local Laplacian filters are heavier and reduce halos, and they can amplify noise along with detail; colorization and matting are not filters at all but sparse solves of the same kind as Poisson, and they need adequate constraints (Chapter 10.3, Edge-preserving optimization — colorization). Reach for this family to process within regions without bleeding across edges. Watch its two characteristic artifacts: banding from coarse range sampling, as in a coarse grid, and gradient reversal, overshoot near strong edges under aggressive detail manipulation; the guided filter and local Laplacian filters were designed to reduce both. The local-global and filter-optimization axes cut through this family: the plain bilateral is a local filter, colorization a global optimization, and the guided filter sits between them with a local linear model.
Seam optimization is discrete optimization, and the algorithm follows the shape of the boundary (Chapter 10.9, Seam optimization). A path constrained to one pixel per row, a carving seam, has an exact dynamic-programming solution; live-wire Mortensen & Barrett 1995 finds general shortest paths. A binary labeling with a submodular energy has an exact graph-cut solution Boykov & Jolly 2001; it needs a data term, which interactive tools obtain from seeds or strokes and automatic ones from a model. Normalized cuts use a spectral relaxation that is approximate and heavier. Reach for this family when the answer is where to put a boundary. It fails on busy images with no low-energy path: when every cut crosses something salient there is no seam to hide, and seam carving distorts visibly. In every case the optimum is optimal for the energy, which need not match the visual result the user wanted.
The methods compose. A photomontage combines a graph-cut seam with a Poisson blend (Agarwala et al. 2004; Chapter 10.9, Seam optimization, Chapter 10.8, Poisson image editing): the graph cut L10.5 decides where to switch between sources, routing the boundary along edges, and the gradient-domain blend L10.4 reduces the remaining differences along it. Cut along the edges, then reconstruct across them. That pipeline is the part in miniature: the three relationships to an edge, used together.
10.10.3 A worked chooser⧉
Use the following task distinctions when choosing a starting point.
- Paste or clone an object and kill the seam. Reconstruct from edges: Poisson seamless cloning, with the healing brush discussed as a related historical formulation; consider a smooth-correction approximation and validate its error (Chapter 10.8, Poisson image editing).
- Compress an HDR scene without flattening local contrast. Either reconstruct from edges (gradient-domain HDR: attenuate the large gradients, re-integrate in log) or smooth except across edges (bilateral base/detail decomposition, compress the base); the part offers both routes (Chapter 10.8, Poisson image editing, Chapter 10.1, Bilateral filtering).
- Denoise while keeping edges crisp. Smooth except across edges: bilateral as a simple baseline, NL-means Buades et al. 2005 for repeated patches or LARK Takeda et al. 2007 for a local oriented-regression model (Chapter 10.1, Bilateral filtering, Chapter 10.2, Non-local means, Chapter 10.7, Locally adaptive regression kernel (LARK)).
- Propagate sparse user input (scribbled colors, a few depth or matte values) to the whole image. Edge-preserving optimization: colorization Levin et al. 2004 / the matting Laplacian Levin, Lischinski & Weiss 2008 (closed-form matting), a sparse solve whose local prior and constraints control propagation (Chapter 10.3, Edge-preserving optimization — colorization).
- Transfer one image's structure onto another's smoothing: flash/no-flash or upsampling a low-resolution result. The joint/cross bilateral Petschnigg et al. 2004 and joint bilateral upsampling Kopf et al. 2007 (joint bilateral upsampling) (Chapter 10.1, Bilateral filtering); or, for a single fast linear model, the guided filter (Chapter 10.6, Guided image filtering).
- Resize, select an object, or stitch a composite. Choose a task-specific path or boundary: seam carving Avidan & Shamir 2007 (resize), intelligent scissors or graph cut (select), graph-cut seam plus Poisson blend (stitch) (Chapter 10.9, Seam optimization).