12.10 Face tracking🔗⧉

Face landmarks let an overlay follow the eyes or mouth through video. Detection finds a face region; landmark estimation locates features; tracking links estimates over time. Recognition asks a different question, identity. These stages can share a model without becoming the same task.

12.10.1 Detecting the face🔗⧉

Before you can analyze a face you must find it. The historic breakthrough is Viola–Jones (Viola & Jones 2004): a cascade of simple Haar-like features, made fast by the integral image (a summed-area table Crow 1984 that lets any rectangle's sum be read in four lookups) and selected by AdaBoost, with a cascade of ever-stricter classifiers that rejects most of the image in the first few cheap tests. It put real-time face detection in every point-and-shoot camera of the 2000s, and it is still the textbook example of "cheap features + boosting + early rejection." Modern detectors are neural: MTCNN (Zhang et al. 2016 (MTCNN)) runs a three-stage cascade of small convolutional networks that jointly detect and roughly landmark the face, and RetinaFace (Deng et al. 2020 (RetinaFace)) is a single-stage dense detector that returns a box, five keypoints, and a confidence in one pass, robust to scale, pose, and occlusion in the wild. In practice, you use library implementations.

12.10.2 Landmarks: pinning down the features🔗⧉

Landmark layouts are model conventions, not universal standards. A 68-point layout is common in classical systems. Legacy MediaPipe Face Mesh Kartynnik et al. 2019 (Face Mesh) uses 468 points; the documented Face Landmarker task returns 478 landmarks and optional expression/transform outputs. Confirm the exact model and indexing before attaching graphics. Predicted depth from one image is not a calibrated scan of the face.

You can run exactly this on your own face (Figure 12.10.2⛶ fullscreen), which makes concrete why the dense mesh is the enabling representation: once every frame carries 478 points in a known order, those points give a stable coordinate system for overlays and tracking.

The landmarks are the correspondence. With a handful of matched points you can align the face by warping it (an affine or thin-plate-spline, the warps of Warping and morphing) into a canonical, upright, fixed-size frame, which is exactly the preprocessing every downstream task (recognition, expression analysis, AR) relies on, because it removes the nuisance variation of pose and scale before the hard work begins.

A box says where to look; landmarks say which part is which. In a fixed layout, an index refers consistently to an eye corner, nose point or other defined location, allowing an overlay or a canonical face crop to use the same indexing in successive frames. Kazemi and Sullivan's regression-tree landmark estimator is a classical example Kazemi & Sullivan 2014. Sparse named landmarks and a dense predicted surface serve different applications, so choose the representation before writing the downstream warp.

12.10.3 Tracking across time🔗⧉

A tracker can initialize the next estimate from the previous frame and re-run detection when confidence falls. Fast motion, occlusion and multiple faces require recovery and identity association. Temporal smoothing reduces jitter at the cost of lag; measure both, and do not let a smooth trajectory conceal a lost track.

12.10.4 Lifting to 3-D: the morphable model🔗⧉

The dense mesh hints at the richest representation: a 3-D morphable model (Blanz & Vetter 1999) describes any face as a low-dimensional combination of basis shapes for identity, expression, and pose, fit to the image so that the projected model lands on the observed landmarks. Once fit, you have a fully parametric face you can re-pose, re-light, transfer expressions onto (the engine behind avatars and many filters), or analyze for gaze and blendshape coefficients. This is the same morphable-model idea developed in Chapter 11.4, Morphable models, here driven by tracked landmarks.

12.10.5 Recognition, and the dark side🔗⧉

Fitting a face model estimates shape or pose; tracking associates observations over time; recognition compares identity. The dominant approach learns an embedding (a vector per face such that two photos of the same person land close and different people land far apart), trained with a margin loss like ArcFace (Deng et al. 2019 (ArcFace)); identification is then a nearest-neighbor lookup in embedding space. The same machinery that powers phone unlock and photo-library grouping also powers surveillance and, in its generative cousin, deepfakes, including face swapping and reenactment driven by the detect-landmark-track-model pipeline above, which is why forgery detection and authentication get their own treatment in Image Forensics and Chapter 41.2, Authentication and provenance (C2PA).

12.10.6 Which library to use optional🔗⧉

For a maintained task interface, start with the MediaPipe Face Landmarker documentation and its supported model bundle. The older toolkits are still useful, each for a different role. dlib provides a CPU face detector and the 68-point regression-tree landmark model Kazemi & Sullivan 2014, and the Python face_recognition package wraps it for quick prototypes. OpenCV includes face detectors, a landmark (Facemark) API and a DNN module that runs exported models. InsightFace collects face detection and ArcFace-style recognition models Deng et al. 2019 (ArcFace); check the license of its pretrained weights before using them outside research. Choose by coordinate conventions, pose/occlusion behavior, platform support and measured end-to-end latency. Identity recognition needs separate embeddings and threshold evaluation; landmark geometry alone is not an authentication system.