11.4 Morphable models🔗⧉

A single linear morph turns one face into another by interpolating a hand-drawn correspondence: it knows about exactly two faces and the straight line between them. Suppose instead you have a hundred faces, all registered to one another, and you ask a bigger question: what does the space of all of them look like, and can I synthesize, recognize, or edit a face by navigating that space? That is the move from a morph to a morphable model, and it is one of the most influential ideas in this part: a whole family of shapes (faces, bodies, a class of objects) captured as a mean plus a basis of deformations, so that members can be approximated by low-dimensional coordinate vectors. The basis also represents implausible shapes; a prior and validity checks restrict how it is used.

The construction is short to state and is built from machinery you already own. Put a training set of examples into dense correspondence (every example's nose-tip maps to every other's), align away the nuisance pose with Procrustes, and take the principal components (PCA) of the aligned shapes. PCA hands back a mean shape and an ordered basis of eigen-deformations, the directions in which the family varies most, and any instance is written

the mean plus a weighted sum of eigen-modes, with coordinates $\mathbf{c}=(c_1,c_2,\dots)$ measured in units of each mode's standard deviation $\sigma_k$. Read in this part's language, $\bar{\mathbf{s}}$ is a mean shape and the $\mathbf{e}_k$ are a learned warp basis over it: each eigen-mode is a small canonical deformation of the mean, and an instance is the mean warped by a linear combination of them. A morphable model is morphing, with the correspondence field constrained to the subspace the data taught us is plausible (Figure 11.4.1⛶ fullscreen).

The same idea recurs at three scales of fidelity, and they share one skeleton. In 2-D, Cootes, Edwards and Taylor's Active Shape / Active Appearance Models (ASM 1995 Cootes et al. 1995 (ASM), AAM 2001 Cootes et al. 2001 (AAM)) build PCA models of landmark shapes (and, for AAMs, of the shape-free texture too) and fit them to images. In 3-D, Blanz and Vetter's 3D Morphable Model of faces (1999) Blanz & Vetter 1999 is the landmark result: a PCA model of 3-D face geometry and texture, fit to a single photograph by analysis-by-synthesis to recover shape, albedo, pose, and lighting at once. The same recipe gives SMPL for human bodies (Loper et al. 2015) Loper et al. 2015 (SMPL) and FLAME for faces (Li et al. 2017) Li et al. 2017 (FLAME). Egger et al.'s 2020 survey Egger et al. 2020 is the map of the whole territory. We develop the skeleton once; the differences are scale (2-D vs 3-D), what gets a basis (shape, texture, expression), and how the prior is parameterized (linear PCA vs a neural net).

11.4.1 Step 1: dense correspondence, the precondition for averaging shapes🔗⧉

Everything rests on a precondition that is easy to state and hard to satisfy: before you can average a hundred faces or take their PCA, every face must be in dense correspondence with every other, so the same anatomical point (the tip of the nose, the corner of the left eye, a specific spot on the cheek) must occupy the same index in every example's data vector. Only then does "average the noses" mean averaging noses rather than averaging a nose in one image with a cheek in another. This is the exact lesson of Chapter 11.3, Morphing, align before you blend, promoted from two images to a whole population: you cannot meaningfully average, or compute a covariance over, points that do not correspond.

For 2-D models the correspondence is often a fixed set of landmarks placed on every training image (eye corners, nostrils, jaw points) by hand or by a detector; an instance is then the concatenation of its landmark coordinates, $\mathbf{s} = (x_1,y_1,\dots,x_K,y_K)$. For dense models (a 3-D face mesh, a body) every vertex must correspond across the set, which is itself a registration problem: fit a common template mesh to every scan so that vertex $j$ lands on the same anatomical point in all of them. Blanz and Vetter's original 3DMM obtained this dense correspondence between laser scans with an optical-flow-style algorithm; modern pipelines use template-fitting and non-rigid registration. The quality of the whole model is bounded by the quality of this correspondence: if it is sloppy, the "mean face" is a blurred average of misaligned parts, and the eigen-modes mix true variation with registration error.

Figure 11.4.1⛶ fullscreen shows the 2-D case on real photographs: MediaPipe's face mesh puts the same numbered point on the same feature of every face, and that is what lets each photograph be warped onto one common shape.

11.4.2 Step 2: Procrustes alignment, factor out pose so PCA sees shape🔗⧉

Two photos of the same face at different positions, scales, and in-plane rotations have different landmark coordinates, yet they are the same shape. If we ran PCA on the raw coordinates, the first and largest eigen-modes would be wasted describing pose: global translation, rotation, and scale, the nuisance variation we do not want the model to spend its budget on. The fix is generalized Procrustes analysis: rigidly align every example (translate to a common centroid, rescale to a common size, rotate to best match) to a shared reference, iterating the reference as the running mean until it stops moving. What remains after Procrustes is shape in the technical sense, geometry with similarity (translation, rotation, uniform scale) divided out, and that is what we feed to PCA. (Pose itself is not discarded; it becomes a few explicit parameters bolted onto the model at fit time, alongside the coordinates $\mathbf{c}$.) The discipline is the same one that opened the DOF ladder of Chapter 11.1, Warping: separate the global rigid/similarity part from the non-rigid part, and model only the latter statistically.

On the 987 photographs of the model in Figure 11.4.1⛶ fullscreen, Procrustes tightens every landmark's cloud; what it cannot remove, a head turned or tilted out of the image plane, stays in the data for PCA to find.

11.4.3 Step 3: PCA, a mean and a basis of eigen-deformations🔗⧉

With the population corresponded and aligned, PCA does the rest. Stack the $N$ aligned shape-vectors as rows, subtract the mean shape $\bar{\mathbf{s}}$, and take the eigenvectors of the covariance (in practice via the SVD of the centered data). The eigenvectors $\mathbf{e}_1,\mathbf{e}_2,\dots$, ordered by decreasing eigenvalue, are the eigen-deformations (for faces, evocatively, the "eigenfaces" of shape), and each carries a standard deviation $\sigma_k$, which measures how far the family typically moves along it. Any instance is

a mean plus a short ($M$-term) weighted sum (Figure 11.4.1⛶ fullscreen). Three properties make this the workhorse of the whole field:

• Low-dimensional. The eigenvalues fall off fast: a handful of modes capture most of the variance, so a face that lived in a few hundred raw coordinates is well described by perhaps a few dozen $c_k$. The family really is a low-dimensional manifold sitting inside a high-dimensional coordinate space, and PCA finds the flat that best approximates it.
• Interpretable, ordered modes. Because the modes are sorted by variance, the first ones capture the biggest axes of variation in the data, later ones finer detail. What those axes are depends on the data, not on what we expect of faces. In the 987 photographs of Figure 11.4.1⛶ fullscreen, the first shape mode follows the head tilting up or down, the second mixes tilting with turning, the third the head turning, and the fourth a smile, while the first appearance modes are skin lightness and light from one side. A collection of faces in varied poses spends its leading modes on pose and expression before face shape. Sweeping a standardized coefficient $c_k$ from $-2$ to $+2$ traces that one axis of facial variation while everything else is held at the mean (the rows of Figure 11.4.1⛶ fullscreen). The modes are not guaranteed to be human-namable, but the leading ones usually are, even when they name something other than identity.
• A built-in prior. A coordinate vector with small $\|\mathbf{c}\|$ (each standardized $c_k$ near zero) is a typical member of the family; a vector far out in the tails is implausible. This is the statistical prior the model buys you: it is not merely a parameterization but a probability over shapes (a Gaussian when that additional distributional assumption is adopted), and that is what later lets fitting regularize toward plausible faces and reject nonsense.

The figures here show a real model, learned from 987 photographs: the mean is the genuine average, and the modes shown are the actual leading eigenvectors of that population's covariance, so what each mode does is what the data produced, not a hand-drawn cartoon.

11.4.4 Step 4: shape and appearance are separate bases🔗⧉

A face is a shape and also a coloring: skin tone, the dark of the brows and lips, shading. A morphable model handles these with two separate bases, and keeping them separate is exactly the separate-shape-from-color discipline of Chapter 11.3, Morphing, now applied to a whole family. Geometry gets the shape PCA above. Appearance (texture) gets its own PCA, but the textures are first warped to the mean shape so they are compared in a common, shape-free frame (every face's texture sampled at the same canonical landmarks before averaging). The result is a mean texture $\bar{\mathbf{t}}$ and an appearance basis $\mathbf{a}_k$, and a full instance is the pair

with two coordinate vectors: $\mathbf{c}$ for geometry and $\mathbf{b}$ for appearance. To render an instance you synthesize the texture in the shape-free frame, then warp it onto the synthesized shape (the warp engine of Chapter 11.1, Warping, the morphable model's biggest customer after morphing): texture-on-the-mean, deformed to the geometry the shape coordinates request (Figure 11.4.1⛶ fullscreen). Why separate them? Because they describe different properties: a change in width need not change skin tone, and a smile is primarily a geometric change. Separate bases do not establish statistical independence, and because untangling them is what makes the model useful: relighting edits the appearance/shading without touching identity geometry; re-posing edits geometry without re-painting the texture; recognition can compare two faces in shape alone, robust to lighting. Active Appearance Models combine the two into a single joint appearance model; the 3DMM keeps shape, texture, and the imaging parameters (pose, illumination) explicit so each can be read off after fitting.

11.4.5 Step 5: fitting a new photo, analysis-by-synthesis🔗⧉

A model is only useful if you can fit it to a new image: recover the coordinates of the face in this photo. The principle is analysis-by-synthesis (equivalently, inverse rendering or render-and-compare): rather than read the parameters off the image directly, search for the parameters whose synthesized image best reproduces the input. Concretely, hold a current guess of the coordinates $\mathbf{c},\mathbf{b}$ and the imaging parameters (pose, camera, lighting); render the model under them; compare the render to the target and measure the residual (the mismatch, in landmark positions and/or pixel colors); and update the parameters to shrink it. Iterate until the residual stops falling. It is the morphable model's answer to the same loop that drives every fitting problem in the book: propose, render, compare, correct (Figure 11.4.1⛶ fullscreen).

A fit combines an observation residual with priors. For example,

Here $R$ renders the model, $\theta$ contains camera and lighting parameters, and $\mathcal V$ is the evaluated visible region. Standardized coefficients make the quadratic terms compatible with an assumed unit-Gaussian prior. PCA itself supplies a basis and variances, not proof of that distribution. The nonconvex fit can confuse shape, illumination and albedo; hidden surfaces are constrained by the model rather than observed in the photo.

Figure 11.4.1⛶ fullscreen also lets us play with the model directly: its sliders move along the modes, a random face is a draw from the prior, and two faces can be morphed in coordinates.

A single photograph may show one cheek clearly and leave the other in shadow or behind the head. The image residual can constrain the visible cheek but gives much less evidence about the hidden one. A morphable model couples them through the learned shape coordinates; a prior discourages extreme coordinate values where the data are weak. The resulting complete face is therefore partly measured and partly supplied by the model. A good image fit does not prove the hidden geometry is correct, and a model built from an unrepresentative population can supply the wrong completion.

11.4.6 Step 6: once fitted, edit by moving in the space🔗⧉

Fitting makes the model useful. Once a photo is reduced to a point $\mathbf{c}$ (plus appearance $\mathbf{b}$, pose, and lighting), every downstream operation is a move in the space. The learned coordinates provide useful directions for editing, but large steps or combinations unsupported by the training data can produce implausible faces. Re-pose / re-illuminate: change the explicit pose and lighting parameters and re-render, giving the same identity from a new angle or under new light, the bread and butter of 3DMM-based face manipulation. Interpolate / average: the straight line between two fitted points is a morph that never leaves the family. The correspondence is built into the basis, which avoids the misaligned-pixel averaging of a cross-dissolve, though an intermediate shape can still be implausible (this is the morph of Chapter 11.3, Morphing, with the correspondence supplied by the model rather than drawn). Edit semantically: add a multiple of a mode that correlates with an attribute (older, wider, more of a smile) to push the face along that attribute while holding the rest. Recognize: compare faces in the identity coordinates, where pose and lighting have been factored out, so the same person under different conditions lands at nearly the same point, the use that motivated much of the original work.

This last group surfaces the identity-vs-expression factorization. The geometry of a face is the identity (bone structure, who it is, roughly constant for a person) composed with an expression (smile, frown, blink: momentary). A flat shape PCA blends the two; the more capable models give them separate factors (FLAME, for instance, carries distinct identity, expression, and pose/jaw parameters) so you can swap one person's expression onto another's identity, or animate a fixed identity through expressions. SMPL does the analogous split for bodies: a shape space (the person's build) and a pose space (the skeleton's articulation), with pose-dependent deformations layered on. The general pattern is to parameterize the deformation in separate groups (identity, expression, pose) each its own little basis, which is the rigging artist's blendshapes with a statistical prior.

11.4.7 Step 7: from linear PCA to neural priors🔗⧉

PCA supplies a linear subspace; paired with a Gaussian coefficient prior, instances are linear combinations of the basis, and the prior is a single ellipsoid. That is its strength: interpretable, cheap, a closed-form mean and modes, and its ceiling. A linear basis cannot capture nonlinear variation (the manifold of faces is curved; large expressions and extreme shapes bend off the flat subspace), it ties detail to how well correspondence and the Gaussian assumption hold, and it tops out at a certain realism. The modern arc keeps the morphable-model idea, a mean and a learned space of deformations you fit and then navigate, but replaces the linear basis with a neural one:

• Regression networks that map a photo straight to 3DMM coordinates in one forward pass, trained with a render-and-compare (differentiable-rendering) loss, which is analysis-by-synthesis with the search amortized into a network. This is how monocular face reconstruction became fast and robust (cross-ref Part 36, 3D and depth).
• Nonlinear / neural morphable models that learn the shape and appearance decoders themselves (autoencoders, implicit fields), so the "basis" is a nonlinear function of a latent code rather than a fixed set of eigen-vectors, which are more expressive than PCA at the cost of interpretability.
• Generative latent spaces (GANs and diffusion models) as implicit morphable models: a face GAN's latent space is a learned, navigable manifold of faces, where interpolation and attribute edits are again "moves in the space," now over a larger (but less labeled) coordinate system (cross-ref Chapter 7.3, Deep learning).

The constant across the whole arc is the framing this chapter began with. Whether the basis is a handful of eigenfaces or the latent space of a billion-parameter network, a morphable model is a mean object plus a learned space of deformations, fit to data by analysis-by-synthesis, and used by moving through that space, a learned warp basis over a mean shape. PCA was the first, clearest instance; the priors got stronger, the idea did not change.

11.4.8 Recap and significance optional🔗⧉

Three ideas outlast the specific models. First, a class of shapes can be modeled as a low-dimensional space: corresponded, aligned and reduced by PCA, a family collapses to a mean plus a basis of eigen-deformations, and a member within the model's range becomes a short coordinate vector. Second, separate shape from appearance (and, further, identity from expression from pose): independent factors, each its own basis, recombined by warping texture onto geometry, the morphing discipline at population scale. Third, fit by analysis-by-synthesis with a prior: search for the coordinates that reproduce the image, regularized toward the model's plausible region, so it can fill in what a single photograph does not show. A linear basis is a modeling choice, not evidence that the population is Gaussian, and a point in the space is plausible only near the data it was built from.

Those three put the morphable model at a crossroads in the book. Backward, it is morphing constrained to a learned family (Chapter 11.3, Morphing) built on the warp engine (Chapter 11.1, Warping). Forward, its analysis-by-synthesis fitting is the template for monocular 3-D face and body reconstruction (Part 36, 3D and depth); its data-term-plus-prior objective is the classical face of the regularized inverse problems that recur in Chapter 14.1, What can super-resolution actually do?; and its linear basis is the ancestor of the neural priors and generative latent spaces of Chapter 7.3, Deep learning, the same idea, a learned space of deformations over a mean, scaled up. The same pattern holds to the end: find the coordinate map, then move the pixels. Only here the map is learned from a population, and moving through it is how you synthesize, recognize, and edit an entire visual class.