42.6 Photography programming on phones⧉
Most photographs are now taken on phones, and most of the computational photography in this book reaches people through a phone's camera application. A developer who wants to ship a new capture method, a better night mode, a document scanner or a burst denoiser, meets the phone through its imaging APIs. Those APIs decide what a program can control, what it can read back, and which parts of the vendor's own pipeline it can use. The recurring surprise is how little of that pipeline is exposed: an application can usually get the finished photograph or the raw frames, and rarely anything in between.
42.6.1 Android: camera2 and CameraX⧉
Android offers two levels Android camera2 and CameraX. camera2 is the low-level API, descended from the Frankencamera's request model (Chapter 42.1, Programmable and modular cameras): the application opens a camera device, configures output streams (a preview surface, a JPEG or YUV stream, a raw stream), and submits capture requests carrying exposure time, sensitivity, focus distance, flash and white balance settings, receiving with each frame a result that records what was applied. Raw frames arrive as 16-bit mosaics that the platform can wrap into a DNG file with the camera's calibration metadata (Chapter 45.14, DNG: the Digital Negative).
camera2 gives the most control and demands the most care. Devices differ in which capabilities they report (manual sensor control, raw output, burst capture, per-frame control), in which stream combinations they accept, and in the delay between a request and the frame it affects. CameraX, a higher-level library, handles the lifecycle and the device differences for common tasks: preview, still capture, video and image analysis. It also offers extensions, night, HDR, bokeh and others, which call the phone maker's own implementation of those modes when the device provides one.
Beneath both sits the camera hardware abstraction layer, which each phone maker implements for its sensors and image signal processor. This is where the vendor's tuning lives, and the API sees it only through the capabilities it chooses to report.
42.6.2 iOS: AVFoundation, Core Image, and the photo library⧉
On iOS, capture goes through AVFoundation Apple AVFoundation and Core Image. A capture session connects camera inputs to outputs; a photo output accepts per-capture settings, including bracketed exposures, and delivers processed images, Bayer raw frames as DNG, or Apple ProRAW. ProRAW is a linear DNG that already contains the result of Apple's multi-frame processing, demosaicked and denoised, with the tone mapping stored as editable metadata rather than baked in. It is not the sensor's samples; it is a processed image in raw clothing, useful for editing and less useful for an algorithm that needs individual frames (Chapter 1.4, What programming language for computational photography?).
Core Image is Apple's image processing library: a graph of filters, evaluated lazily and on the GPU when a result is requested, with color management built in. Vision provides detection and segmentation, faces, text, people, the subject of a photograph, running on the device's neural engine. PhotoKit gives access to the user's library, and its edits are non-destructive in the sense of Chapter 8.12, Recap ISP, non-destructive editing: the original is kept and an edit is stored as adjustment data beside it, so another application can revert or continue it.
42.6.3 The closed middle⧉
The pipelines that make phone photographs good, HDR+ and Night Sight on Pixel phones (Chapter 17.6, HDR+ and burst imaging on phones), Deep Fusion and Night mode on iPhones, run inside the camera stack and are largely closed to third parties. An application can ask for their output, sometimes through CameraX extensions or a quality setting on iOS, but cannot change their parameters, feed them its own frames, or read their intermediate results: the aligned burst, the merged raw before tone mapping, the segmentation masks used for local adjustment.
That leaves two choices. Take the vendor's finished image and process it further, inheriting every decision it made, including its denoising and tone mapping (Chapter 43.2, Why phones are so good (at photography)). Or take raw frames and reimplement the pipeline, which gives full control and full responsibility: alignment, merging, demosaicking, color, tone mapping, all within the phone's time and power budget, and usually without the vendor's per-sensor tuning. Research prototypes take the second path; most shipping applications take the first.
42.6.4 Running the computation on the device⧉
A phone has several processors that can run image code: CPU cores, a GPU, a digital signal processor, and a neural accelerator (Chapter 29.6, Hardware backends: GPU, NPU, DSP). The GPU is reached through Metal on iOS and Vulkan or OpenGL ES on Android; learned models through Core ML on iOS and LiteRT (formerly TensorFlow Lite) on Android, each of which chooses among the available processors (Chapter 29.7, Efficient neural network inference).
Two practical constraints dominate. Memory: a burst of twelve 12-megapixel raw frames is several hundred megabytes before any intermediate, and the operating system terminates applications that use too much. Heat: a phone sustains far less power than it can reach briefly, so a pipeline that runs in 200 ms on a cool phone may take twice that after a minute of use. Measure on the device, over sustained use, with the capture running, rather than on a desktop or a single cold run.
A cross-platform application can share its numerical core, written in C++ or compiled from Halide, between Android and iOS, and write capture and display separately on each. Chapter 1.4, What programming language for computational photography? discusses that split; the interfaces between the shared core and each platform's buffers, strides and lifetimes are where most of the integration bugs appear.