42.5 Networking and image transport🔗⧉

A photograph taken on a phone this morning has probably already been copied several times: to a cloud library, to a messaging service that re-encoded it, to a web page that serves it at five sizes in two formats. Images are among the largest things most networks carry, and moving them raises two questions. The first is how to deliver an image efficiently: at the right size, in the right format, starting to appear before the whole file has arrived. The second is where to compute on it, on the device that holds it or on a server with more power, when the cost of computing somewhere else is the cost of moving the pixels there.

42.5.1 Delivering the right image🔗⧉

A 24-megapixel photograph shown in a 400-pixel-wide slot on a phone wastes nearly all its bytes. Web delivery therefore serves several renditions of each image and lets the client pick: an HTML srcset lists widths and the browser chooses by its layout and pixel density, and content negotiation lets a server answer the same request with a newer format when the client's request says it can decode it. Photo services generate thumbnails and previews at upload so that browsing a library never touches an original.

Each rendition is a downsampling, and every server-side resize service is a resampling problem (Chapter 6.5, Resampling and upsampling). A resize that subsamples without prefiltering aliases, producing moiré on fabrics and jagged edges on architecture, and a resize applied to gamma-encoded values darkens fine high-contrast detail (Chapter 8.2, What the numbers mean). These errors are invisible in a test on a smooth portrait and obvious on a photograph of a window screen.

A file can also be made to arrive usefully in pieces. A progressive JPEG orders its coefficients so the first bytes decode to a coarse version of the whole image, refined as the rest arrive, rather than a sharp image that loads from the top down. JPEG XL was designed with progressive decoding and reversible recompression of existing JPEG files among its goals Alakuijala et al. 2019 (JPEG XL). The codecs themselves, WebP, AVIF AVIF, HEIF ISO/IEC 23008-12 (HEIF) and JPEG XL, are compared in Chapter 8.11, File formats and compression.

Containers matter as much as codecs. HEIF, the container phones commonly use, can carry more than one image: a depth map, the frames of a burst, a live photo's video, a gain map for HDR display (Chapter 17.7, Encoding HDR images). A pipeline that transcodes such a file to a single JPEG discards those auxiliary images, often without anyone noticing.

42.5.2 Tiles and pyramids over the network🔗⧉

A gigapixel panorama or a museum's scan of a painting cannot be downloaded to be viewed. Web viewers borrow the editors' solution (Chapter 42.4, Photoshop-style editors): store the image as a pyramid of tiles and fetch only the tiles the current view needs at the current zoom. The IIIF Image API standardizes the request, so a client asks for a region, a size and a rotation of an image by URL and any compliant server answers IIIF Image API. Cloud-optimized GeoTIFF does the same inside one file: tiles and overviews are laid out so that a client can read the few byte ranges it needs over HTTP instead of the whole file. Satellite archives are served this way.

The pyramid has to be built with the same care as a thumbnail, since each level is a prefiltered downsampling of the one below (Chapter 6.6, Pyramids and wavelets).

42.5.3 Where to compute🔗⧉

It is tempting to treat a data center as unlimited free computation and to send every hard image problem there. The computation is rarely the expensive part. A full-resolution photograph is several megabytes; uploading it and downloading a result costs time on the radio, battery, the service's bandwidth bill, and a round trip that a local processor does not pay. And a photograph that never leaves the device cannot be leaked, subpoenaed or used for training.

Phones therefore compute on the device whenever they can. Burst merging, tone mapping and portrait segmentation run on the phone's own processors, which were designed for this work (Chapter 17.6, HDR+ and burst imaging on phones; Chapter 29.6, Hardware backends: GPU, NPU, DSP). Large learned models push the other way: a generative edit that needs a model of billions of parameters may only be practical on a server, and phone makers now split features between the two, with on-device models distilled from server ones.

The same lesson governs a processor's memory (Chapter 29.2, Modern CPUs: memory hierarchy, parallelism, and what it takes to go fast) and the editors of this part, which place their module boundaries to avoid writing intermediate images (Chapter 42.3, Lightroom-style raw developers).

42.5.4 Sending a recipe instead of the image🔗⧉

When a server must do the work, the device can still avoid shipping full images both ways. Transform Recipes Gharbi et al. 2015 (Transform Recipes) observe that most photographic enhancements, tone and color adjustments, local contrast, even some stylizations, change an image smoothly with respect to the input: within a small region, the output is well approximated by a simple function of the input pixel, such as an affine color transform plus a luminance curve.

The device uploads a heavily downsampled and compressed version of the photograph. The server runs the expensive enhancement on it and, instead of returning an image, fits the recipe: for each block of the image, the parameters of the local transform that map input to output. The recipe is much smaller than an image, and the device applies it to its own full-resolution original, so fine detail the server never saw is preserved. Upload and download both shrink, and the device's work is a cheap per-pixel evaluation.

The idea recurs. HDRnet Gharbi et al. 2017 predicts the same kind of local affine coefficients with a network on a low-resolution input and applies them at full resolution on the device, removing the server altogether, and CompPhotoRoom's recipes are the same division of labor between a small description and a local evaluation (Chapter 42.3, Lightroom-style raw developers). The recipe works when the operation is locally smooth in the input. It fails for operations that create content, inpainting or generative fill, where the output at a pixel has no simple relation to the input there; those must return pixels.