42.1 Programmable and modular cameras⧉
Suppose you want to capture an HDR bracket on a phone: three frames, one stop apart, as fast as the sensor allows. On a camera that exposes only a shutter button, you cannot. You can ask for "HDR mode" and receive whatever the vendor's pipeline produces, but you cannot say which exposures, you cannot know which frame used which settings, and you never see the frames themselves. Most of the multi-image methods in this book, burst denoising, exposure merging, flash/no-flash, focal stacks, need exactly that missing control.
A programmable camera exposes the capture pipeline to software: exposure time, gain, focus, white balance and flash can be set per frame, and each returned frame says what was actually used. This chapter follows that abstraction from a research platform to the phone APIs, looks at how people opened cameras that were never meant to be programmed, and then at the hardware counterpart, cameras built from swappable modules.
42.1.1 What a closed camera hides⧉
A conventional camera runs its control loops, autoexposure, autofocus, auto white balance, inside firmware the user cannot reach, and exposes their results through a few modes. That is the right product for most photographers and the wrong instrument for research. Three things are missing.
1/ Per-frame control. Changing exposure between consecutive frames, at sensor rate, is what a bracket or an HDR viewfinder needs. A mode dial changes settings for the next photograph, not the next frame.
2/ Ground truth about each frame. An algorithm that merges frames must know each frame's exposure time and gain, not the value it requested. Sensors apply new settings with a delay of one or more frames, and a request that arrives mid-readout may land on the frame after.
3/ The frames themselves. A finished JPEG has been demosaicked, denoised, tone mapped and compressed (Chapter 8.12, Recap ISP, non-destructive editing). A merge needs raw or at least linear samples.
The section What programming language for computational photography?#Programmability, or the lack thereof makes the practical point: choose a platform by the measurement it lets you take.
42.1.2 The Frankencamera: a camera as a pipeline of requests⧉
The Frankencamera Adams et al. 2010 (Frankencamera) defined the abstraction that capture APIs still use. Its authors built two platforms, a custom camera assembled from off-the-shelf parts and a modified Nokia N900 phone, and an API called FCam that ran on both.
FCam models the sensor as a pipeline that accepts shots and returns frames. A shot is a complete description of one capture: exposure time, frame duration, gain, white balance, the region of interest, and actions scheduled against the exposure, such as firing the flash at a given time or moving the lens. The program pushes shots into the pipeline; the sensor consumes them in order, several in flight at once, and each shot is applied exactly to its own frame. Each frame comes back tagged with the parameters actually used, the timestamps of its exposure, and the state of the other devices at that time: where the lens was, whether and when the flash fired.
Two consequences matter for algorithms. First, a bracket is a list of shots, so it runs at full sensor rate with no dead frames between settings. Second, because each frame carries its own parameters, the merge code reads exposure from the frame rather than trusting the request, which removes a whole class of silent errors. The paper demonstrated this with applications that needed it, among them an HDR viewfinder that alternates exposures and a capture mode that uses the gyroscope to keep the sharpest frames of a handheld burst.
The Frankencamera also made a methodological point. Computational photography research had relied on tethered DSLRs and hand-written scripts, and could not test an idea at viewfinder rates. A programmable camera turned "shoot a burst and process it later" into something that could run live, on the device, which is where phone photography went next (Chapter 17.6, HDR+ and burst imaging on phones).
42.1.3 From FCam to camera2⧉
Android's camera2 API, introduced with Android 5.0 in 2014, adopted the same request and result model Android camera2 and CameraX. An application builds a CaptureRequest (exposure time, sensitivity, frame duration, focus distance, flash mode, output streams), submits it once or as a repeating request for the preview, and receives a CaptureResult per frame with the values the hardware actually applied and the sensor timestamp. A burst is a list of requests submitted together. Underneath, the camera hardware abstraction layer that phone makers implement is itself request based, so the model runs from the application down to the driver.
The model reached phones; full control did not reach every phone. camera2 reports a hardware level per camera and a list of capabilities, and manual sensor control, raw output and per-frame control are capabilities a device may lack. The practical rule is the one Chapter 42.6, Photography programming on phones repeats: query what the device supports before designing a capture around it.
A programmable camera does not make the vendor's own pipeline programmable. The same phone can expose per-frame manual control through camera2 while keeping its night mode, its HDR merge and its tone mapping closed. The API gives you the raw ingredients or the finished dish, and rarely the steps between.
42.1.4 Opening cameras that were not meant to be opened⧉
Dedicated cameras mostly remained closed, and communities opened some of them from the outside. CHDK CHDK (Canon Hack Development Kit) loads alongside the firmware of many Canon compact cameras and adds scripting, raw capture on models that lacked it, and exposure bracketing beyond the stock limits. Magic Lantern Magic Lantern does the same for Canon EOS cameras; on some bodies it added raw video recording and a dual-ISO mode that reads alternate sensor lines at two gains, recovering dynamic range in a single exposure.
These projects show both how much capability sits unused in a camera's hardware and how fragile access through reverse engineering is: support is per model, per firmware version, and ends when the volunteers move on. On the open-hardware side, single-board computers such as the Raspberry Pi pair an interchangeable sensor module with an open camera stack, libcamera, that exposes per-frame control in the FCam tradition libcamera and Picamera2. The sensors are modest, but everything from the register settings to the raw readout can be inspected.
42.1.5 Modular cameras⧉
Programmability opens the software; modularity opens the hardware. The idea is to treat the camera as a set of swappable parts, sensor, lens, processing, rather than a sealed unit, so that one part can be upgraded or specialized without replacing the rest.
Interchangeable lenses are the long-standing example, and medium-format cameras have long accepted interchangeable digital backs that contain the sensor. Google's Project Ara (2013–2016) pushed the idea into the phone Project Ara. An Ara phone was a frame of standardized slots, and modules, including the camera, attached through a common electrical and data interface that let a module be added or removed while the phone ran. A user could in principle swap in a better sensor, a telephoto or an infrared module. Ara was canceled in 2016 before shipping. Other attempts shipped in narrower forms: lens-style cameras that clip onto a phone and use it as the viewfinder over a wireless or wired link (Sony's QX series, the Olympus Air, the DxO One), and snap-on modules for one phone family, such as a zoom camera module for Motorola's Moto Z.
The trade-off is plain once stated. A connector between sensor and processor adds size, cost, power and latency, and a camera module needs its own sealing, optics and calibration. An integrated phone places the sensor millimeters from the processor over a dedicated high-bandwidth interface and calibrates the pair at the factory. Phones therefore stayed integrated, and flexibility moved into software: the modular part of a modern phone camera is the capture API and the processing that follows it.