1.4 What programming language for computational photography?🔗⧉

Use Python to learn the algorithms and explore images. Its array and machine-learning libraries perform their expensive operations in compiled code, often on a GPU. Use C++ or Halide when you need more control over an imaging pipeline's runtime and memory. JavaScript and WebAssembly provide a way to deliver interactive tools in a browser; phone applications also need platform-specific capture APIs.

The book provides Python and C++ implementations of the same ideas. A language does not determine speed by itself. In the course, replacing naive Python pixel loops with optimized compiled code sometimes produced speedups near 1000×. Such a comparison measures implementation choices as well as language: array operations, fusion, memory layout, and parallelism matter. Measure the complete pipeline, including image transfers and startup, before choosing what to optimize (Part 29, Making computational photography code fast).

1.4.1 On the desktop: Python, C++, and Halide🔗⧉

Python is a convenient default for numerical experiments. NumPy represents images as arrays, while PyTorch and JAX add GPU computation and automatic differentiation. A per-pixel loop in Python can be slow; an array expression delegates the loop to compiled code. Vectorization can also create large temporary arrays, so memory traffic still matters. In conventional CPython builds, the global interpreter lock limits simultaneous execution of Python bytecode; compiled libraries can release it, and optional free-threaded builds have different behavior. Check the interpreter and extensions you deploy (Python documentation).

C++ offers explicit control of data layout, allocation, and parallel execution. That is useful for a pipeline with a tight latency or memory budget, but it requires more care with indexing, ownership, and build configuration. Standard containers and resource-owning objects reduce the need for manual allocation. Static types catch some errors; they do not catch a wrong color space or an out-of-bounds access automatically.

Halide separates an image-processing algorithm from its schedule: where and in what order values are computed, stored, vectorized, and parallelized. This lets the same algorithm support several execution strategies without rewriting its mathematical definition. Good schedules can avoid intermediate images and repeated memory transfers. They do not guarantee a fixed speedup over well-optimized C++ (Halide).

Much of the computational-photography literature was prototyped in MATLAB. For years it was the platform for this kind of work: a strong numerical and linear-algebra library, and an interactive IDE that could read, display, and manipulate images natively: an image was just a matrix, imread/imshow and array slicing made experiments effortless, and the Image Processing Toolbox supplied the rest. NumPy carries the same array-as-image approach into Python, while frameworks such as PyTorch combine tensor operations with automatic differentiation and GPU execution. We mention it because older papers and code you will meet are written in it, and the array-thinking it taught carries over directly to NumPy. Choose the environment that lets you reproduce and inspect a method, rather than translating working research code only for consistency of language.

For example, NumPy's out = a + 0.5*b makes a point operation almost as direct to write as its mathematical definition. That short edit-and-run cycle matters when the algorithm itself is uncertain. Once the operation is settled, C++ can give more control over storage and execution, with a heavier build cycle and more responsibility for object lifetimes. Halide adds a scheduling language when the performance question is where to compute and retain intermediate values. These are different costs to pay at different stages of a project.

1.4.2 In the browser: JavaScript and the web stack🔗⧉

A browser makes an experiment easy to share: the reader opens a page and changes the parameters. JavaScript can manipulate pixels through Canvas 2D; WebGL and WebGPU support GPU processing, while WebAssembly runs compiled code. Workers can move long computations off the interface thread, though they do not automatically make an algorithm faster.

Keep numerical intermediates in floating-point arrays or textures. The usual Canvas 2D pixel interface uses 8-bit, sRGB-encoded RGBA values, so repeated reads, processing, and writes can lose precision. Decode the transfer function for operations that model the addition or averaging of light, and encode for display. Some editing operations deliberately use nonlinear representations; Chapter 4.3, Encoding: linear, gamma or log explains the distinction.

Camera access requires a secure context and user permission. Cross-origin image access also needs appropriate server headers; a tainted canvas throws a security error when its pixels are read. Test the formats, GPU features, and image sizes on the browsers you intend to support. Libraries such as OpenCV.js and ONNX Runtime Web can supply vision or inference operations, but check which operations and execution backends their browser builds include.

The names are worth knowing because this book's own interactive figures use them. Canvas 2D's getImageData and putImageData expose pixels as an RGBA Uint8ClampedArray, enough for point operations and small filters on a thumbnail. Typed arrays such as Float32Array hold the floating-point working buffers. getUserMedia opens the camera, with facingMode choosing the front or rear lens on a phone; several demos in this book use it. OffscreenCanvas lets a Worker draw without the page, and devicePixelRatio tells you how many device pixels a CSS pixel covers, which is the number to scale the canvas backing store by. For 3-D and GPU rendering, three.js is the common WebGL engine, and the book's optics and cover scenes are built on it. OpenCV.js is OpenCV compiled to WebAssembly; TensorFlow.js and ONNX Runtime Web run neural networks in the page, on the GPU where the browser allows; MediaPipe provides real-time face, hand and pose tracking, which the face-landmark demo uses. These interfaces evolve, so check current documentation before depending on a detail.

Keep track of where the pixels live. A CPU buffer, a canvas, and a GPU texture are different storage locations, and moving an image among them can cost more than a simple filter. A Worker can keep computation off the UI thread without making the arithmetic itself faster. WebAssembly provides compiled CPU execution; shaders provide parallel GPU execution. Choose the mechanism to match the operation and include upload and readback in the timing.

Repeatedly writing an intermediate into an eight-bit buffer also quantizes it repeatedly. Keep a floating-point working buffer when a sequence of operations needs more precision, and encode and quantize when preparing the display output. Distinguish the canvas's CSS size from its backing-store resolution: a high-density screen can display several device pixels per CSS pixel. A mismatch can make a correct rendering look soft.

1.4.3 On the phone: Android and iOS🔗⧉

Separate capture from processing. On iOS, AVFoundation supplies camera capture, Core Image supplies image operations, and Metal supports custom GPU work. On Android, CameraX provides higher-level capture while camera2 exposes lower-level controls. The available manual controls, raw formats, frame rates, and combinations of streams depend on the device and camera. Query capabilities before designing a burst around them (Android camera metadata).

A raw file does not always mean untouched sensor samples. Apple ProRAW combines computational photography with a linear DNG representation and editing metadata; it has already undergone processing. Bayer RAW and ProRAW have different uses. Request and inspect the format your algorithm needs rather than inferring it from the DNG extension (Apple's ProRAW overview).

For an application on both platforms, a shared C++ processing library can reduce duplicated numerical code. Capture, display, and GPU integration still require platform-specific work. Another reasonable choice is to use each platform's image libraries when they already provide the needed operations. Measure sustained latency, memory, and energy on the device: a short benchmark may hide thermal throttling. Chapter 42.6, Photography programming on phones covers these choices in more detail.

For example, a burst-alignment routine can operate on arrays in a shared processing library while the platform-specific code acquires frames and supplies timestamps, exposure metadata, and buffer layouts. The numerical routine can then be tested on saved captures without opening a camera. The capture wrapper still determines whether the frames are raw mosaics or processed RGB, whether buffers need copying, and when they remain valid. Separating those responsibilities makes it easier to distinguish an algorithmic error from a capture or integration error.

For example, a shared C++ image-processing core can be called through JNI on Android and through a suitable C++ bridge on iOS. The application still has to translate platform image buffers, handle their lifetimes and strides, and coordinate capture and display. Sharing the mathematics does not make these interfaces identical.

Decide early whether the application processes the camera's rendered photographs or replaces part of the capture pipeline. A rendered photograph inherits the vendor's denoising, tone mapping, and color decisions. Raw access gives more control over those operations, but also makes the application responsible for them. Access to individual frames and their exposure metadata matters for a burst algorithm; access to a finished JPEG alone does not supply those measurements.

1.4.4 Libraries🔗⧉

Implement a core operation once to understand its assumptions, then use established libraries for a larger project. Pillow handles common image-file operations; NumPy and SciPy provide numerical arrays and algorithms; OpenCV and scikit-image provide image-processing and vision routines. PyTorch adds tensor computation and automatic differentiation. C++ counterparts include OpenCV, Eigen for linear algebra, and LibRaw for raw decoding.

Library defaults are part of the computation. Check channel order, integer versus floating-point arithmetic, transfer functions, coordinate conventions, and boundary handling. A function named “resize” does not tell you whether its output matches the sampling convention assumed by another library. Chapter 9.1, Developing, Testing and Debugging gives small inputs that make such differences visible.

1.4.5 Vibe coding🔗⧉

An LLM can help draft image I/O, visualizations, and numerical functions. Check the result against inputs with known answers: a constant for normalization, an impulse for a filter kernel, or a half-plane for resampling. A plausible photograph can conceal a wrong transfer function, offset, or boundary rule. See Chapter 9.1, Developing, Testing and Debugging.

The failures are not confined to difficult algorithms. While reproducing Photobios, the morphing slideshow that ages a person across a photo collection (Part 28, Photo collections), the coding agent handled face alignment and a dynamic-programming path through the images, then repeatedly got the cross-fade wrong. It took about five re-prompts to correct it. The intended operation was:

# start and end: aligned float images in the same chosen color representation
# t: blend fraction between 0 and 1
frame = (1 - t) * start + t * end

At zero the output must equal the start image, at one it must equal the end image, and equal inputs must remain unchanged. These simple checks establish properties that visual plausibility alone does not.

The book's interactive figures were developed with coding agents and often needed several rounds of correction. Review the computation and the displayed result together: a correct kernel can still be shown with the wrong range or encoding. A model's explanation of its code is not an independent check of that code.

1.4.6 Programmability, or the lack thereof🔗⧉

Running code inside a camera and processing files from that camera are different capabilities. Many dedicated cameras provide raw files, exposure bracketing, or remote control while keeping their onboard processing closed. Those files are sufficient for many of this book's experiments. Methods that require precise per-frame control or access to a raw burst place stronger demands on capture.

CHDK CHDK (Canon Hack Development Kit) and Magic Lantern Magic Lantern extend selected camera models with features such as scripting or additional recording modes. Support is model-specific. Phone APIs provide programmable capture on supported devices, but they do not expose every vendor pipeline or every physical camera configuration (Figure 1.4.1⛶ fullscreen). A Raspberry Pi camera with libcamera/Picamera2 offers another platform for capture experiments; check its control timing and raw formats against the proposed measurement.

Choose the platform from the measurement you need: raw samples or a processed image, exposure and focus control, timing, and synchronization. Then determine where to run the reconstruction. Phone processors offer substantial parallel computation within a tight power budget, but dedicated cameras also use specialized image processing and learned features. Neither category is uniformly open or computationally weak.