25.2 Non-photorealistic rendering🔗⧉

Many restoration methods aim to recover a faithful image: to recover detail the optics blurred away, to denoise, to expose well, to undo haze. Non-photorealistic rendering (NPR) does the opposite, and on purpose. It throws information away to make a photograph more like a drawing or a painting, to communicate rather than reproduce. A medical illustration leaves out the distracting tissue and draws a bold line around the organ that matters; a courtroom sketch gives you the witness's posture and not the wood grain of the bench; a cartoon is flat color and a few black outlines. The value of each is in what was left out.

The important part is that the tools are the same ones we built in Part 10, Edges matter. That is why this chapter belongs in a computational-photography book rather than an art-history one. The operation that makes a good cartoon is flatten the unimportant texture, keep and darken the meaningful edges, and that is exactly the edge-preserving / base–detail decomposition L10.1show, used now to abstract instead of to enhance. The bilateral filter that we used to smooth noise while protecting edges is the very same filter that collapses a face into flat poster-like regions; the edge detector that we used to find structure is the one that inks it. So NPR is less a new field than a new intent placed on familiar machinery, which is also why it makes a fitting coda. The chapter runs from stroke-based painterly rendering (paint the photo with brushes, the course p-set), through edge-preserving abstraction (bilateral + Difference-of-Gaussians), to example-based stylization, and on to the bridge into neural style, where the hand-coded rules give way to a learned prior.

25.2.1 What NPR is for, and the one idea🔗⧉

The goal is a depiction, not a reproduction: a painterly, sketch, or cartoon rendering that emphasizes structure and suppresses clutter. It sits closer to illustration than to photography, and its value lies in what it omits. A photorealistic render is judged by how little you can tell it apart from a photograph; an NPR render is judged by how clearly it says one thing.

The one idea underneath all of it is structure-aware abstraction. Keep what the visual system actually uses to read a scene (the strong edges, the large-scale tone, the silhouette of the subject) and simplify everything else: fine texture, mid-tone noise, busy backgrounds. This is big lesson L40.1show, separate the subject and commit (first placed in Chapter 40.1, Human factors and the art of photography), executed not with a lens aperture or a choice of viewpoint but in pixels, and it is carried out with the L10.1show edge-preserving machinery. The two lessons meet here: deciding what carries meaning L40.1show and having a filter that respects edges while smoothing within them L10.1show are two useful ingredients in the image-abstraction methods discussed here. Other NPR techniques emphasize geometry, materials, or a particular mark-making process.

In practice: a cartoon is flat color regions plus bold outlines. The flat regions come from an edge-preserving smoother (one that reduces mixing across strong intensity differences); the outlines come from an edge detector. That two-part recipe (smooth the insides, ink the borders) covers an important family of cartoon filters, and the rest of this chapter is variations on it plus one older, stroke-by-stroke approach that gets at the same place from the painter's side.

25.2.2 Stroke-based / painterly rendering (and the brush p-set)🔗⧉

The oldest computational route to a painting is to actually paint, to repaint the photograph with discrete brush strokes rather than filter its pixels. The canonical method is Hertzmann's Painterly Rendering with Curved Brush Strokes of Multiple Sizes Hertzmann 1998, and its central move is coarse-to-fine. Start with a blank (or roughly toned) canvas and a heavily blurred version of the photo as the reference. Lay down big, rough strokes to match that blurry reference: broad daubs of approximate color. Then sharpen the reference by one pyramid level and add smaller strokes, but only where the current painting still disagrees with the reference. Repeat down to the finest level. The effect is that detail appears only where it is needed (an eye, a highlight, a busy edge) while flat areas like a sky or a cheek stay loose and impressionistic, painted once with a fat brush and never revisited. (Figure 25.2.1⛶ fullscreen.)

What keeps this from looking like random daubs is the structure built into where and how the strokes go:

• Orientation. Strokes run perpendicular to the image gradient, that is, along the "grain" of the image, following the contour of a cheek or the flow of a sky rather than cutting across them. A stroke laid along an edge reinforces it; one laid across it would smear it. This is the gradient field of Chapter 10.8, Poisson image editing L10.4show used as a direction field for the brush.
• Size equals scale. Coarse strokes are placed against coarse pyramid levels, fine strokes against fine ones, a direct use of the Gaussian / image pyramid (Chapter 6.6, Pyramids and wavelets). The pyramid does double duty: it supplies the blurred references and it sets each layer's stroke size.
• Curved strokes. Rather than a single dab, a stroke can be traced as a long curve that follows the orientation field, bending with the image's flow and stopping according to color error, stroke-length limits, or an unreliable direction in a nearly flat region. Long curved strokes are what make the result read as confident brushwork rather than stippling.
• Error-driven placement. A new stroke is added only where the canvas-versus-reference error exceeds a threshold $T$. This directs effort toward regions still poorly represented at the current scale. Overlapping strokes can disturb earlier work, so it is not a guarantee of monotonic numerical convergence.

The pedagogical point is that the parameters are the style. One algorithm, with a few knobs (the stroke size range, opacity, positional jitter, curvature limit, the blur radius of the references), produces an impressionist, a pointillist, or an expressionist look. Big strokes with high jitter and low opacity give a loose, Monet-ish wash; tiny round strokes with no curvature give pointillism; long high-curvature strokes give something more Van-Gogh-like. Nothing in the algorithm changes; only the brush does. This is the cleanest demonstration of "style as a small parameter set," and it is the conceptual ancestor of the learned methods at the end of the chapter, which replace this hand-chosen parameter vector with a statistic learned from an example.

A related exercise is developed in the course brush problem set: implement the layered, gradient-following brush painter. Build the pyramid of blurred references; place oriented strokes coarse-to-fine, perpendicular to the local gradient, adding a stroke only where the error exceeds threshold; then explore a couple of brush/parameter styles on your own photograph. Comparing several parameter settings on one photograph exposes what the brush model controls. Follow the problem set itself for required functions, inputs, and deliverables. (Figure 25.2.4⛶ fullscreen.)

25.2.3 Edge-preserving abstraction: bilateral + Difference-of-Gaussians🔗⧉

The filtering route reaches a cartoon directly, and in real time, fast enough for live video. The canonical pipeline is Winnemöller, Olsen and Gooch's Real-Time Video Abstraction Winnemöller et al. 2006, and it is the two-part recipe from above made literal:

1. Flatten the regions with an (iterated) bilateral filter. The bilateral reduces smoothing across strong intensity differences L10.1show, so a textured face or a noisy wall collapses into broad, flat, poster-like regions while the boundaries between them stay crisp. Run it two or three times for a stronger, more painterly flattening. Optionally quantize the result, snapping luminance (or color) into a small number of bands, for the hard-stepped, posterized look of a screen-print.
2. Draw the lines with a Difference-of-Gaussians. Blur the image at two nearby scales and subtract: $$ D = G_{\sigma_1} * I - G_{\sigma_2} * I, $$ which responds wherever intensity changes rapidly (i.e. at edges) and is near zero in flat regions. Threshold $D$ and you get black ink edges: a clean line drawing of the scene's structure. (The DoG is the classic band-pass / edge operator; with $\sigma_2$ a bit larger than $\sigma_1$ it approximates a Laplacian-of-Gaussian.)
3. Composite. Lay the thresholded DoG lines on top of the flat, quantized color, and the result is a cartoon (flat regions, bold outlines) produced cheaply enough to run on each frame of a video. (Figure 25.2.2⛶ fullscreen.)

This pipeline matters because it makes the importance of edges explicit. The bilateral step is literally the base/detail filter of Chapter 10.1, Bilateral filtering L10.1show: only here we keep the base (the flattened regions) and discard the detail (the texture we worked so hard to preserve elsewhere). The same affinity-weighted smoothing that protected detail in tone mapping is now used to annihilate it. Same tool, opposite intent, which is exactly the L10.1show recurrence boxed above.

There is one common refinement. Plain isotropic DoG, applied independently at every pixel, gives broken, noisy edges: short dashes and speckle, because nothing ties one edge pixel's response to its neighbor's. Kang, Lee and Chui's flow-based coherent line drawing Kang et al. 2007 fixes this by first computing a smooth edge-tangent flow (an orientation field that says, at each pixel, which way the local edge runs) and then evaluating a DoG across the local tangent direction and smoothing its response along the flow. The result is long, clean, coherent lines, like a confident pen stroke instead of a scratchy pencil. It is the same idea as the painter's oriented, curved strokes, now applied to the edge operator: respect the local orientation and the marks join up. (Figure 25.2.3⛶ fullscreen.)

For video, the remaining problem is temporal coherence: filtered independently, the flat regions and ink lines flicker and crawl frame-to-frame, which is distracting and unnatural. One approach is to make the filtering motion-aware (propagate the abstraction along the optical flow so a region keeps its color and a line keeps its place as the scene moves) while handling occlusions and unreliable motion. Temporal stability does not follow from optical flow alone.

25.2.4 Example-based stylization and the bridge to neural style🔗⧉

The methods so far hand-code what to keep: edges via a DoG, strokes via a gradient field, flat regions via a bilateral. Two final ideas point past that: first toward directing the abstraction by where attention goes, then toward learning the style from an example rather than coding it.

Eye-tracked abstraction (DeCarlo and Santella, Stylization and Abstraction of Photographs DeCarlo & Santella 2002) makes the abstraction non-uniform on purpose. They abstract the image at multiple levels of detail and then keep fine detail only where a viewer actually looked (measured with an eye-tracker) abstracting everything else away. The result reproduces, in the rendering, what the viewer's own visual system did: it foregrounds what drew attention and lets the rest dissolve. This is NPR as directing attention, and it is the most explicit possible link back to L40.1show (separate the subject and commit) with the "subject" defined by literal measured gaze.

Image Analogies Hertzmann et al. 2001 takes the example-based idea to its general form: a single hand-stylized example pair teaches an entire filter, the style inferred from one demonstration rather than coded by hand, the pre-deep ancestor of neural style, the hinge between the classical and learned worlds. The full treatment lives in Chapter 25.1, Style transfer (and it is surveyed as a model in Chapter 7.3, Deep learning); here it is enough to register it as that hinge.

That transition leads directly to neural style. Classical NPR, the whole chapter to this point, hand-codes what to keep: edges, strokes, flat regions, all chosen by the algorithm's designer. Neural style (Gatys, Ecker and Bethge Gatys et al. 2016) replaces those rules with deep-feature statistics, content captured by a convolutional neural network (CNN)'s feature activations, style captured by the correlations among those features (the Gram matrix), and optimizes an image to match both. Feed-forward stylization (Johnson, Alahi and Fei-Fei Johnson et al. 2016) then trains a network to do the same in a single pass, making it real-time. The goal is identical to the painter's (a depiction that keeps content and imposes a look) but the prior is now learned rather than designed L7.6show. The full treatment is in Chapter 25.1, Style transfer and Chapter 7.3, Deep learning; here it is enough to see the lineage.

The chapter's takeaway: across painterly strokes, bilateral + DoG cartoons, eye-tracked abstraction, and neural style, the constant is abstraction guided by structure: decide what carries meaning, keep and exaggerate it, simplify the rest. The methods differ only in how they decide and how they execute. NPR is, in the end, L10.1show + L40.1show turned into a rendering style: the edge-preserving machinery doing the simplifying, and the subject-separation instinct deciding what to spare.

25.2.5 Region-based stylization: stained glass, low-poly, mosaics🔗⧉

Strokes and edges are not the only handle on abstraction. A different family partitions the image into a few large regions and floods each with one flat color: abstraction by tessellation. Choose the tiling to respect structure and the result reads instantly: a Voronoi / centroidal tessellation gives the leaded panes of "stained glass by numbers"; a Delaunay low-poly triangulation seeded at corners and edges gives the faceted look of vector portraits; superpixels (SLIC) give organic patches. The dial is the number and shape of the cells: a few large ones read as a bold poster, many small ones approach the photograph. It is the same thesis as the rest of the chapter (L10.1show, abstraction guided by structure), with "what to keep" carried by region boundaries rather than brush orientation; edge-aware seeding helps align tiles with contours, but exact boundary preservation may require constrained tessellation or segmentation. It is kin to photomosaics (each tile chosen to match the local color) and, when the partition is found by graph-cut labeling, to the seam machinery of Chapter 10.9, Seam optimization.

25.2.6 Artistic screening and halftoning🔗⧉

The oldest abstraction of all is reproducing a continuous tone with discrete marks (the print shop's problem) and it turns expressive once the marks themselves carry meaning. Ordinary halftoning trades resolution for tone with clustered dots, ordered dithering, or error diffusion (Floyd–Steinberg Floyd & Steinberg 1976 scatters each pixel's quantization error to its neighbors, often producing a dispersed-dot texture, though directional patterns and worms can remain). Artistic screening (Ostromoukhov & Hersch) Ostromoukhov & Hersch 1995 replaces the regular dot screen with shaped, image-bearing screen elements, so the halftone microstructure spells a second picture, lettering, or texture: correct tone at arm's length, a deliberate pattern up close. Stippling, engraving/hatching, and ASCII-art are the same move with different mark vocabularies: here the abstraction lives in how tone is quantized into marks, not in strokes, edges, or regions. Its tonal effect relies on limited spatial resolution: the eye spatially averages fine marks into tone (Chapter 3.3, Visual processing); the screen, its frequency, and its angle are the style, and the CMYK screen angles that dodge moiré tie it back to Chapter 4.1, Color space 101.