1.4 What language for computational photography?🔗⧉

The ideas in this book are language-agnostic, but code has to be written in something, and the book ships in two editions — a Python one and a C++ one. We never show both in the same book; you pick the edition that matches what you're trying to do, because the two languages serve different moments. Python is for thinking and prototyping; C++ is for shipping and running fast. The concepts are identical either way.

The gap between those moments is larger than newcomers expect, and it is widest exactly when you write the low-level code yourself — your own convolution, say, rather than a library call. In Frédo's experience that naive pixel loop runs roughly 1000× faster moving from Python to optimized C++, and you can win at least another ~10× on top of that with Halide. So choosing a language is also choosing a performance regime, which is why the same algorithm can feel either instantaneous or hopeless depending on where you run it; we return to squeezing out those factors in Performance engineering and Halide.

Python is the right default for learning, prototyping, and research. Its great virtue is readability: with NumPy, array code reads almost like the math it implements — out = a + 0.5b does what it looks like it does. It is fast to write and explore, with interactive notebooks and instant feedback, and it sits atop a vast ecosystem (NumPy, SciPy, Pillow and OpenCV, scikit-image, Matplotlib, and the entire deep-learning stack — PyTorch, JAX). Memory is managed for you, and it is the lingua franca of research and machine learning. The catch is speed at the pixel level: a naive per-pixel Python loop is orders of magnitude slower than C, so you must vectorize — push the loop down into NumPy's compiled core — or your code crawls. The global interpreter lock (GIL) limits true CPU threading, packaging and deployment are awkward, dynamic typing lets type bugs hide until run time, and you get little control over memory layout. Still, through NumPy and PyTorch it is fast enough* for most of what we do here.

C++ is the right default for performance and production. It is fast and predictable — close to the metal, with control over memory layout, cache behavior, and single-instruction-multiple-data (SIMD) vectorization — and it is what real camera ISPs, real-time pipelines, and shipped products are actually written in. Static types catch errors at compile time, and it deploys cleanly to phones and embedded hardware. The cost is real: it is verbose and slower to write, manual memory management is a rich source of bugs (out-of-bounds accesses, leaks, use-after-free), the build and toolchain story is heavier, and the edit–compile–run loop is far less interactive than edit–run. Once the algorithm is settled and it has to run fast, on device, at scale, that productivity tax is worth paying.

There's a third option for the hot paths. Halide is a domain-specific language, embedded in C++ and Python, for high-performance image processing. Its central idea is to separate the algorithm — what each pixel computes — from the schedule — how to tile, vectorize, parallelize, and fuse that computation across the memory hierarchy. You get near-hand-tuned speed without hand-writing the tangled, hardware-specific loops, and the same algorithm retargets to CPU, GPU, or DSP by swapping the schedule. It is the basis of real camera stacks. The cost is another language and mental model, best reserved for hot paths you have already identified — premature scheduling is its own trap. We cover it properly in Performance engineering and Halide.

MATLAB deserves a historical note, because much of the computational-photography literature was prototyped in it. For years it was the platform for this kind of work: a strong numerical and linear-algebra library, and an interactive IDE that could read, display, and manipulate images natively — an image was just a matrix, imread/imshow and array slicing made experiments effortless, and the Image Processing Toolbox supplied the rest. It has since been largely displaced by Python — NumPy fills the same array-is-an-image niche, free and open — and especially by PyTorch, whose automatic differentiation and GPU / deep-learning ecosystem MATLAB never matched. We mention it because older papers and code you will meet are written in it, and the array-thinking it taught carries over directly to NumPy.

A note on libraries. This book has you implement the core algorithms from scratch — a bilateral filter, a homography solve, a Poisson blend — because writing one yourself is how you actually understand it (the reference implementations in imageops exist for exactly that). But once a technique is clear and you want to build with it — or dive straight into an advanced project — reach for mature libraries instead of reinventing the plumbing. In Python, the workhorses are Pillow (PIL) for loading, saving, and basic operations, NumPy / SciPy for array math, OpenCV and scikit-image for a large catalogue of vision and image routines, and PyTorch for anything learned (and as a GPU array + autodiff engine even when no network is involved). In C++, the counterparts are OpenCV again (the de-facto standard), Eigen for linear algebra, LibRaw for decoding camera raw, and Halide for the performance-critical pipelines above. The rule of thumb: implement once to learn, then stand on the libraries to go far.

Finally, a note on vibe coding — writing image code with the help of a large language model (LLM). It is now genuinely part of the toolkit: great for boilerplate, file I/O, visualization, and explaining a baffling error, and a quick way to draft a function or a test. But image code is numerically and perceptually subtle, and "looks plausible" is emphatically not "correct." A resampling kernel that is subtly wrong, a color conversion off by a gamma, an antialiasing filter that rings — these produce images that look fine until you check them. So keep the model on a short leash for filtering, resampling, color, and antialiasing, and always verify against a hand-checkable input — a constant, an impulse, a half-plane — before you trust it. And don't assume the failures will be confined to the hard parts: AI coding sometimes fumbles the trivial even when it nails the difficult algorithm above it — see the cross-fade anecdote in the "Vibe coding" chapter. Treat generated image code as guilty until tested (see Developing, Testing and Debugging).