21.4 Generative inpainting🔗⧉

Every method so far answers the same question: given everything around the hole, what most plausibly belongs inside it? The answer improves as the prior gets stronger, from the neighbors to the rest of the image to a database to a learned model, but the question does not change. Text-conditioned generative fill changes the question. You are no longer asking what belongs there. You are saying what goes there, and the model's job is to make your answer fit the photograph.

That is a different tool with a different set of ways to be wrong, which is why it gets its own chapter even though it sits on top of the ladder.

21.4.1 Filling a hole against saying what goes in it🔗⧉

Mechanically the change is small. One implementation uses the mask-constrained sampling loop introduced in the inpainting overview: re-inject known pixels at each step and let the denoiser generate the rest. Dedicated inpainting networks can instead receive the mask and masked image as conditioning; not every system uses the same loop. What is new is that the denoiser takes a second conditioning input, a text embedding, so the prior it imposes is not "typical images" but "typical images matching this description". The hole is filled from the intersection of what the surroundings imply and what you asked for.

The practical consequence is that failure moves. A patch-based method can fail by choosing source content that fits locally but makes the wrong object or structure globally. A text-conditioned method fails by producing exactly what you asked for in a way that does not belong: right object, wrong scale, wrong light, wrong shadow direction. The model is very good at the object and much weaker at whether the object could have been photographed there, because the object came from the prompt and the plausibility has to come from the image.

Outpainting is the same operator with the mask moved outside the frame. It is harder for a reason that has nothing to do with the model: an interior hole is surrounded by context on all sides, while an extension has less surrounding measured context; a strip added at one edge may touch known content on only one side. The further out you go the less the photograph constrains, until the model is simply generating. Uncropping a portrait one aspect ratio is a fill; uncropping it to a landscape is generation with a photograph pinned to one edge.

21.4.2 Inpainting in a latent space, and the seam it creates🔗⧉

There is a detail here that matters in practice and is easy to miss from the equations. Many diffusion models denoise a compressed latent produced by an autoencoder and decode it to pixels at the end. A factor of eight in each spatial dimension is common in some architectures, but pixel-space models and other compression factors also exist. Inpainting in that latent means the known region makes a round trip through the encoder and the decoder even though nobody asked to change it, and it does not come back identical. The autoencoder is lossy, most visibly on fine texture and on text.

So a naive latent inpaint quietly degrades the whole photograph to fill one hole. A direct way to preserve the known region is to composite: keep the original pixels outside the mask, take the generated pixels inside it, and blend across the boundary. That preserves the pixels outside the blending band exactly, and it introduces the artifact you will actually see, a seam where original detail meets decoded detail. The two sides differ in their high frequencies, so the join is visible at the grain level even when the content matches perfectly.

A refinement pass can reconcile texture across the seam after generation at the model's working resolution. Alternatively, match local color and noise, feather an appropriate boundary band, or use gradient-domain blending. Refining the entire image can alter pixels the user intended to preserve, so restrict the operation when exact preservation matters. A rectangle visible at 100% can come from resolution, color, or noise mismatch; it does not diagnose one missing pipeline step.

21.4.3 What it gets wrong🔗⧉

The characteristic failures each have a name, and a named artifact is one you will see (💡 Chapter 9.2, The expert's eye as evaluation).

Identity drift. Fill part of a face and the reconstructed half is a plausible face, not that person's face. The prior is over faces in general, unless surrounding facial features, reference images, or identity conditioning constrain it sufficiently. Even then, inferred detail is not a verified record of the person.

Almost-letters. Text in a generated region can become convincing letterforms with incorrect spelling or content. Text conditioning and stronger models can improve legibility, but a readable generated sign or plate still need not match the hidden one. Signage and license plates are where this is most obvious and most consequential.

Geometry that does not continue. Straight lines, tile grids, brickwork and horizons enter the hole at one angle and leave at another. Long-range structural agreement across a large mask is exactly what the receptive field of the earlier convolutional inpainters could not span, and a diffusion prior is better at it without being reliable.

Disagreement between runs. Run the same fill twice and you get two different results, both plausible. With different random samples this is expected, because the conditional model admits several completions, and it is the single fact that makes this class of method hard to evaluate.

And the point that inverts the intuition: a stronger prior does not make hallucination rarer, it makes it more convincing. A weak prior fails visibly, blurring or smearing, and you throw the result away. A strong prior fails invisibly, producing something sharp and coherent that was never measured. The better these models get, the more carefully the output has to be treated.

21.4.4 Judging a fill with no reference🔗⧉

For a real object-removal edit the hidden region usually has no available reference, so the full-reference metrics of Chapter 9.3, Image metrics have nothing to compare against. You can hold out a region you actually measured and score the fill against it, which is how the literature reports numbers, but the original patch is only one possible completion of its context. Reference metrics are useful for controlled tests of recovery, while plausibility, edit intent, and preservation need additional evaluation.

What is left is looking, done properly, which is Chapter 9.2, The expert's eye as evaluation's subject. Three things carry most of the weight here. Keep a fixed set of hard masks, the ones that break fills: a face, a hand, text, a tiled floor, a horizon, a large mask with little context. Flip between the original and the fill in place rather than scanning them side by side, because the seam and the geometry break announce themselves as changes and hide as comparisons. And sort by worst case rather than by average, because a fill method ships on its failures, not on its mean.

21.4.5 An inpainted region is not evidence🔗⧉

The forensic consequence deserves stating on its own, because it is the one that leaves the technical domain. A generated region is a sample from a prior conditioned on the rest of the photograph and on a prompt. It is not a measurement, it can reflect constraints from the measured surroundings but supplies no new measurement of the hidden region, and its plausibility is a property of the model rather than evidence about the scene. A generated plate character or facial detail should not be treated as recovered evidence merely because it is sharp. Super-resolution may retain measured constraints, but details selected mainly by the prior require the same caution.

This is why the provenance machinery of Chapter 41.2, Authentication and provenance (C2PA) matters more as these tools get better, and why Image Forensics treats a confident, seamless fill as a harder problem than a clumsy one. The book's position is the one stated at the top of the ladder in Chapter 21.1, Inpainting, texture synthesis, and object removal and worth repeating at its top rung: the more convincing the fill, the more explicitly it has to be labeled.