42.2 Image processing libraries⧉
Nobody ships a hand-written convolution. A production pipeline calls a library, and the library decides things the algorithm description leaves open: how the image is laid out in memory, whether an operation runs now or when its result is needed, on which processor, over the whole image or a tile at a time. Two libraries that both offer "Gaussian blur" can differ by an order of magnitude in speed and by a pixel in where the result sits. This chapter surveys the main families by the design choices that separate them.
42.2.1 What a library provides⧉
At minimum, an image library provides a container (dimensions, channels, sample type, strides), file input and output, and a catalog of operations: point operations, convolutions, resampling, geometric transforms, color conversion, morphology, statistics. Beyond that, the libraries diverge along four axes.
1/ Execution: eager, where each call computes its full output before returning, or lazy, where calls build a description of the computation that runs later, when a result is requested.
2/ Memory: whole images in memory, or tiles and scanlines streamed through a bounded working set.
3/ Processor: CPU, GPU, or both behind one interface.
4/ Extensibility: a fixed menu of operations, or a way to write new ones that run as fast as the built-in ones.
Most libraries also carry conventions that are part of the computation even though no documentation page calls them algorithms: channel order, whether 8-bit values are treated as encoded or linear, where pixel centers sit, and what happens at the border. Chapter 1.4, What programming language for computational photography? gives the practical warning; Chapter 9.1, Developing, testing and debugging gives the small test inputs that expose a mismatch.
42.2.2 Eager whole-image libraries⧉
The familiar libraries are eager and whole-image. OpenCV Bradski 2000 (OpenCV) offers a very broad catalog of classical image processing and vision in C++ with Python bindings. NumPy arrays with SciPy and scikit-image van der Walt et al. 2014 (scikit-image) give a scientific interface where an image is an array and every operation returns a new one. Pillow and ImageMagick concentrate on file formats, conversions and simple raster operations.
The eager model is easy to reason about: every intermediate exists, can be printed, and can be compared with a reference. That makes it the right model for learning and for testing, and it is what the book's reference implementations use. Its cost is memory traffic. A pipeline of ten operations over a 50-megapixel RGB float image writes and reads ten intermediates of 600 MB each, and on a modern processor that traffic, not the arithmetic, sets the running time (Chapter 29.2, Modern CPUs: memory hierarchy, parallelism, and what it takes to go fast). Whole-image libraries also fail outright when the image does not fit in memory.
42.2.3 Demand-driven, streaming libraries⧉
libvips Martinez & Cupitt 2005 (VIPS) takes the opposite approach. A call does not compute anything; it adds a node to a pipeline. When the final output is written, the library pulls small regions of it through the pipeline, and each operation asks its input for only the region it needs to produce its own. A resize of a gigapixel scan therefore never holds the whole image: at any moment only a few scanlines per operation are in memory, and worker threads process different regions in parallel.
This is the demand-driven model that the editors in the next two chapters also adopt, and it has one requirement that eager libraries never face: every operation must be able to say which input region it needs for a given output region. A point operation needs the same region; a blur of radius r needs the region grown by r; an operation such as a histogram equalization needs the whole image before it can produce any output. The library, not the operation, must then decide how to handle the border of the image, since the operation only ever sees a region.
42.2.4 Separating the algorithm from its schedule⧉
Streaming fixes memory; it does not fix the choice of loop order, tiling, vectorization and which intermediates to keep, and those choices dominate performance. Halide (Ragan-Kelley et al. 2012; Ragan-Kelley et al. 2013) splits a pipeline into two parts written separately: the algorithm, a set of pure functions over integer coordinates, and the schedule, which says in what order to evaluate them, what to compute ahead of time, what to recompute, and how to parallelize and vectorize. The same algorithm can then be scheduled for a phone's CPU, a desktop GPU or a DSP, and a schedule change cannot introduce a numerical bug. Chapter 29.3, Halide at a high level: decoupling algorithms from schedules develops the model.
Halide is a compiler rather than a library of operations, and it is used both inside products and as the engine behind other systems: CompPhotoRoom's kernels are Halide compiled to WebAssembly, and gamut uses ahead-of-time Halide kernels for some of its operations.
42.2.5 GPU and tensor libraries⧉
Graphics APIs (OpenGL, Vulkan, Metal) expose the GPU through shaders, and much interactive image processing runs as fragment or compute shaders over textures. Compute platforms such as CUDA add general kernels and libraries tuned for them. Deep learning frameworks, PyTorch Paszke et al. 2019 (PyTorch) and JAX among them, have become general image processors in their own right: a convolution, a resampling or a color transform is a tensor operation that runs on the GPU and is differentiable, which is what makes it possible to optimize an image or a pipeline's parameters by gradient descent.
Their conventions differ from classical image libraries: images are batches of channel-first float tensors, value ranges are whatever the model was trained on, and resampling functions have their own alignment conventions. Moving an image between a tensor framework and a file format is where encoding errors enter, most often an sRGB-encoded image treated as linear (Chapter 8.2, What the numbers mean).
42.2.6 Choosing⧉
Match the library to the job. For learning and testing, an eager array library, with every intermediate visible. For images larger than memory, or servers that process many images, a demand-driven library. For a fixed pipeline that must run fast on a device, a scheduled compiler or hand-written GPU kernels. For learned components, a tensor framework, with an explicit conversion at each boundary. Whatever you choose, keep a slow reference implementation of the pipeline and compare against it on small inputs: the faster the library, the more of the computation is hidden from you.