Lecture Notes 0 — Intro🔗⧉

Hand-designed algorithms, learning, and generative AI🔗⧉

Hand-designed methods specify how to reconstruct an image. Learned restoration estimates that mapping from examples. Generative methods learn a distribution of images and can sample possible outputs, often conditioned on a capture. These approaches coexist: a camera may use a conventional geometric alignment stage, a learned denoiser, and a hand-designed tone curve.

The distinction does not by itself tell us how faithfully a result follows the data. A hand-designed prior can erase real texture; a learned model can be constrained by the measurement. Generative restoration is especially useful when many plausible images explain the input, but the selected detail need not be the detail that was actually there. Evaluation must reflect whether the task is measurement, photographic rendering, or synthesis.

Rich Sutton's The Bitter Lesson argues that general methods able to exploit growing computation tend to outperform elaborate hand-coded knowledge over time. Natural-language processing gives the standard example. For decades, parsing a sentence meant hand-building grammars that encoded linguists' knowledge of syntax; statistical parsers trained on corpora of annotated sentences overtook them, and neural models later did so with still less hand-designed structure. The linguistic knowledge did not stop being true. It stopped being the thing programmers wrote down directly, and moved into the data, the annotations and the choice of what to measure. Imaging offers an important place for physical knowledge within learned methods: we can model projection, blur, sensor noise, and the color-filter mosaic to generate training pairs or constrain reconstruction. Those pairs are exact for the simulator, not necessarily for a real camera. Calibration errors, clipping, motion, and unmodeled processing can make synthetic training data misleading. Chapter 7.2, Machine learning and Chapter 7.3, Deep learning develop these approaches.

This also changes an expectation that developed around photographs. Perspective construction, photographic capture, and documentary practice all helped establish the image as evidence about a particular scene. Photojournalism gave special importance to the idea of recording an actual instant. That ideal was never a complete description of image-making: staging, framing, and manipulation remained. Generative synthesis makes the distinction especially visible because a convincing image need not originate in a corresponding event. The question is what relationship to an event a viewer is entitled to expect.

The physical knowledge accumulated in imaging still has work to do. A hand-designed denoiser puts assumptions directly into its algorithm; a learned denoiser can inherit them through simulated noisy/clean pairs, its architecture, or a measurement constraint. Knowledge of the camera has moved into a different part of the design. The practical question is where that knowledge is most useful, and how closely the simulated measurements represent the real capture.

A multidisciplinary field🔗⧉

Computational photography draws on optics and sensor engineering to form measurements, algorithms and machine learning to reconstruct images, and perception and photographic practice to judge the result. Performance engineering determines what can run on a device. Ethics matters when the processing changes how we represent a person or an event.

A few neighboring fields overlap with computational photography so heavily that it helps to say exactly how they relate and differ.

Computer vision is the closest. A useful way to carve up vision is by its three ingredients: 2-D images (the pixels themselves: filtering, features, matching), 3-D (the geometry behind the pixels: depth, shape, pose), and meaning (semantics: what objects and scenes the pixels depict). Computer vision ultimately cares most about the last two: it treats the image as a means to recover 3-D structure or to understand content. Computational photography inverts that emphasis: its product is the 2-D image, a better photograph. But it freely reaches into 3-D and meaning when they help the picture: depth to fake a shallow-focus background, segmentation to brighten a face and not the sky, recognition to pick the frame where everyone is smiling. The same tools, pointed at a different goal.

Computer graphics runs the arrow the other way, synthesizing images from models of geometry, materials, and light. Computational photography borrows its rendering and image-based-rendering machinery (and, increasingly, shares it outright: gradient-domain editing, light fields, differentiable and neural rendering all live in both). Image processing is the classical signal-processing bedrock the whole enterprise is built on: filtering, transforms, sampling, and compression. Models of capture help choose the processing. And optics supplies the physics of image formation, which computational photography increasingly co-designs with the computation rather than treating as fixed (coded apertures, lensless cameras, end-to-end optimized lenses).

Because the field straddles these neighbors, so does its literature. A great deal of computational photography is published at the major computer-vision conferences (CVPR, ICCV, and ECCV) and at the graphics venue, SIGGRAPH (and SIGGRAPH Asia / ACM Transactions on Graphics). ICCP, the IEEE International Conference on Computational Photography, is a venue dedicated to the field.

These disciplines appear together inside a single feature. Take a phone's night mode. It leans on physics (catching scarce photons), electrical engineering (the sensor that catches them), algorithms (aligning and merging a burst of frames), AI (learned denoising), perception (tone-mapping the result so it reads naturally), performance engineering (finishing within a short capture budget on a phone without excessive energy use), and ethics (how much to "beautify" a face, and whether to tell the user), all at once. Each discipline constrains the others: a more expensive reconstruction can require fewer input frames or a longer wait for the picture.

Vibe coding🔗⧉

An LLM can help draft image I/O, visualizations, and numerical functions. Check the result against inputs with known answers: a constant for normalization, an impulse for a filter kernel, or a half-plane for resampling. A plausible photograph can conceal a wrong transfer function, offset, or boundary rule. See Chapter 9.1, Developing, Testing and Debugging.

The failures are not confined to difficult algorithms. While reproducing Photobios, the morphing slideshow that ages a person across a photo collection (Part 28, Photo collections), the coding agent handled face alignment and a dynamic-programming path through the images, then repeatedly got the cross-fade wrong. It took about five re-prompts to correct it. The intended operation was:

# start and end: aligned float images in the same chosen color representation
# t: blend fraction between 0 and 1
frame = (1 - t) * start + t * end

At zero the output must equal the start image, at one it must equal the end image, and equal inputs must remain unchanged. These simple checks establish properties that visual plausibility alone does not.

The book's interactive figures were developed with coding agents and often needed several rounds of correction. Review the computation and the displayed result together: a correct kernel can still be shown with the wrong range or encoding. A model's explanation of its code is not an independent check of that code.

An image is an array🔗⧉

A color image is stored as a grid of channel values, commonly red, green, and blue (Figure 8.1.1⛶ fullscreen). These are samples or measurements; drawing enlarged pixels as squares is a display convention, not a complete model of sensor integration.

More formally, an image is a function defined over a 2-D domain, the grid of integer positions $(x, y)$, whose values live in a range, here the space of colors. We store that function as a three-dimensional array of size $\text{height} \times \text{width} \times \text{channels}$ and write $I(x, y, c)$ for the value at column $x$, row $y$, channel $c$. A grayscale image has a single channel, and we drop the index and write $I(x, y)$.

Usually a pixel carries exactly three channels, but nothing forces that. Some images add a fourth (for transparency, which we return to under alpha below) and scientific hyperspectral images keep dozens or hundreds of narrow wavelength bands instead of three broad ones. One very practical trap follows from this freedom, and it bites beginners constantly: a Portable Network Graphics (PNG) file may carry an alpha channel, so a file you assume is three-channel RGB hands you four channels instead. If the rest of your code expects three, every pixel downstream is misread. The fix is trivial once you know to look (check the channel count on load), but the symptom (colors scrambled, code not crashing) is hard to diagnose if you do not.

How the array sits in memory🔗⧉

The array is two- or three-dimensional, but computer memory is one long line of numbered addresses. Something has to decide how the grid gets flattened into that line, and there is no single right answer. Different ecosystems made different choices, which matters the instant you pass an image between them.

The examples use packed planar C++ storage and HWC Python arrays. In contiguous planar storage, the element offset is $cWH+yW+x$; in contiguous interleaved storage, it is $(yW+x)C+c$ (Figure 8.1.6⛶ fullscreen). A NumPy array can have other strides after slicing or transposition, and tensor dimension order does not uniquely determine physical memory layout. Query shape, strides, and contiguity at library boundaries. Row strides are $W$ elements per planar channel or $WC$ elements for interleaved RGB, before padding; byte strides also include element size.

The second decision is the order within a pixel. Most of the world, and this book, store the channels as red, green, blue (RGB), but a few libraries, OpenCV being the notorious one, use BGR. Mix the two and red and blue swap silently: skies turn orange, skin turns blue, and nothing throws an error. So always know which order your data is in.

Why care about any of this for everyday code? Mostly you should not. The mental model is just "a grid of pixels," and the accessor we build below hides the packing entirely. Two situations force you to think about it: performance (a loop that strides through memory in contiguous order runs far faster than one that hops around, so the right layout for an algorithm matters when speed does), and interoperability (handing an image to a library that expects a different layout means rearranging the array first). Outside those, the packing is an implementation detail.

Walking contiguous elements usually makes better use of cache lines and vector instructions. For packed HWC data, scalar loops run y, then x, then c; for planar CHW, c, then y, then x. A kernel that processes one whole pixel at once can hide the channel loop. Tiling, vectorization, compiler transformations, and hardware affect the best schedule; measure the implementation (Figure 8.1.7⛶ fullscreen).

Planar storage makes same-channel values contiguous. Interleaving keeps all channels of each pixel together, which can help operations that use them jointly. Vector implementations can handle either layout. Conversion costs and the whole pipeline matter more than declaring one layout universally faster.

This book carries a concrete, working C++ image data structure; seeing it once makes the planar arithmetic above concrete. It is deliberately minimal: three integers for the shape, and a single flat buffer of floats in planar order:

struct Image {
    int width, height, channels;
    std::vector<float> data;        // planar: c*width*height + y*width + x

    Image(int w, int h, int c)
        : width(w), height(h), channels(c),
          data(static_cast<size_t>(w) * h * c, 0.0f) {}

    // The accessor: hides the index arithmetic, so the rest of the
    // code can think in plain (x, y, c).
    float& operator()(int x, int y, int c) {
        return data[static_cast<size_t>(c) * width * height + static_cast<size_t>(y) * width + x];
    }
};

This minimal accessor assumes validated nonnegative dimensions, a checked allocation-size product, and in-range indices. Use a sufficiently wide unsigned type for offset multiplication. A production image class also needs const access and explicit boundary reads; an unchecked mutable reference cannot implement zero padding safely. In Python, the book uses arr[y, x, c].

The accessor makes the storage choice concrete without spreading it through the rest of the program. A call such as img(x,y,0)=1.0f looks like indexing a three-dimensional image, while the function computes one offset into the flat buffer. Changing packing later requires changing this mapping and the code at library boundaries, rather than every mathematical expression. Read access beyond an image edge should be a separate policy from obtaining a mutable reference to a stored pixel.

The loop-order effect comes from reuse of fetched data. A cache line brings several neighboring values into the processor together. Traversing a packed row uses those values before moving on; jumping one full row per inner iteration can touch many cache lines while using only one value from each. A different schedule may recover that reuse through tiling or vectorization, which is why the benchmark measures the implemented loop rather than proving a universal ordering rule.

Test on inputs you can verify by hand🔗⧉

A tiny input lets you compare every output value against a calculation independent of the implementation. Use several sizes, including a non-square image, odd dimensions, and a single row or column. These expose transposed axes, vector tails, and boundary assumptions that a convenient square image misses.

• A constant tests the expected gain. A normalized blur with constant-preserving boundaries should return that constant; a derivative should return zero. Zero padding deliberately darkens the border of a nonzero constant, so decide whether that is the specified behavior before calling it a bug. For color code, use different constants in the three channels to expose channel swaps.
• An impulse tests how a local input spreads. For a linear, shift-invariant filter away from boundaries, the output gives the convolution kernel. For a general linear operator it gives one column of the operator, which may differ at another location. Test center and boundary impulses separately.
• A step edge exposes ringing, halos, shifts, and color fringes. Try both orientations and a diagonal.
• A rectangle adds corners and unequal extents. It is useful for checking coordinate conventions and differences between horizontal and vertical processing.

Apply these tests to individual stages. Compare integer results exactly when the specification requires it. For floating-point results, choose tolerances in the working units, accounting for rounding and changes in reduction order. One common comparison is

The absolute tolerance handles values near zero; the relative tolerance allows error proportional to magnitude. Check nonfinite values before this comparison. A large tolerance can conceal an algorithmic error, so report the maximum difference and its location as well as pass/fail.

Apply the same discipline inside the pipeline: feed these inputs to your intermediate stages, not only to the final program. A failed whole-pipeline impulse test tells you that something is broken; running each stage on an impulse tells you which stage. The synthetic inputs and the binary-search habit are the same idea from two directions: small inputs make each stage checkable, and checking each stage localizes the bug.

Crashes and bounds🔗⧉

Out-of-bounds accesses are common in neighborhood operations: a neighbor at the left edge has a negative column, or the last row asks for a sample at height. In C++, an invalid read or write is undefined behavior. A write may corrupt adjacent memory and trigger a crash much later. Invalid reads, dangling pointers, and data races can also fail without a useful message.

Check indices at the point of access, use a debugger, and enable the compiler's memory-error diagnostics during development. Assert properties that the algorithm actually requires, such as valid extents or a nonzero normalization denominator. Do not assert that every intermediate lies in $[0,1]$: many linear filters and color transforms legitimately produce values outside that range.

Debug assertions help catch internal mistakes, but they do not replace input validation in a release build. Reject unsupported dimensions and formats at the interface. For intended accesses outside an image, apply the specified extension rule (repeat, mirror, wrap, or zero) before reading memory. An assertion catches an unhandled access; it does not choose the correct boundary condition.

Put a check where the assumption is first used. For example, a resampler can produce an invalid source coordinate long before an eventual crash in an unrelated allocation. Checking the coordinate immediately before the load identifies the operation that broke the contract. Similarly, after computing a normalized filter, check that its weights sum to the expected value; after a transform, check finiteness before sending the result to the next stage. The first failed invariant is usually more informative than the final wrong image.

Keep internal development assertions separate from input validation. A release build may disable assertions, but it still needs to reject unsupported external sizes, formats, and indices safely. Removing a debug check must not make malformed input legal.

C++ for Python programmers🔗⧉

The problem sets in this book are written in C++, and if Python is your home language this appendix is the bridge. You are not expected to arrive knowing C++: the early psets ramp you up, and what follows is the map of what changes.

Why C++ at all? It is compiled and close to the metal, so the per-pixel loops that image processing lives on can run far faster than equivalent scalar Python loops; the gain depends on the workload and compiler, and is not a comparison with optimized NumPy kernels. It is still the standard where performance matters, in graphics and imaging, and it is the language Chapter 29.4, Halide programming plugs into for the last chapters. The price is bluntness: more typing, fewer guardrails, and memory you sometimes manage yourself. The good news is that for this book you stay in a small, safe corner of the language.

What changes, coming from Python🔗⧉

• Compiled, not interpreted. You translate the whole program to machine code once, with the compiler, then run the resulting binary. A whole class of mistakes (a misspelled name, a type mismatch) is caught at compile time rather than when the line happens to run.
• Static types, declared up front. Every variable has a fixed type, written before its name: int n = 5;, float x = 0.5f;. You cannot later rebind n to a string. Functions spell out the type of each argument and of the return value.
• Semicolons and braces. Statements end in ;, and blocks are delimited by curly braces { }, not by indentation. Indent for readability; the compiler ignores it.
• Headers and implementation files in the course code. A header (.h) declares what exists; an implementation (.cpp) defines it. You #include the header, you do not import.
• Value semantics by default. Assigning or passing an object by value generally copies it, though moves and copy elision can avoid copies, unlike Python, where a name is a reference to a shared object. You ask for sharing explicitly, with a reference.
• A leaner toolkit. Fewer built-in higher-order conveniences; you write more loops by hand. These exercises do not use a C++ array library to do the work for you, so the loop over pixels is yours to write (and the compiler's to make fast).
• Memory is explicit, but mostly hidden here. There is no garbage collector, yet if you stick to local variables and std::vector you almost never call new or delete.

A first taste, side by side🔗⧉

Brighten an image by a constant factor, written as a function in each language. In Python the body is a single NumPy broadcast:

def brighten(im, factor):
    return im * factor

Assume floating-point NumPy inputs, with import numpy as np. In C++ with the course's Image class it is an explicit loop, and every difference above shows up at once:

Image brighten(const Image &im, float factor) {
    Image out(im.width(), im.height(), im.channels());
    for (int c = 0; c < im.channels(); c++)
        for (int y = 0; y < im.height(); y++)
            for (int x = 0; x < im.width(); x++)
                out(x, y, c) = im(x, y, c) * factor;
    return out;
}

Types on everything, the loop written out, const Image &im to pass the input by reference without copying it or letting the function change it, and semicolons throughout. The vectorization is gone, but the compiled loop is fast.

A few more comparisons🔗⧉

The same algorithm three ways (the C++ clamp example needs C++17 and <algorithm>): plain Python with the explicit loops C++ makes you write, the C++ itself, and the vectorized NumPy one-liner. The shading marks what C++ adds over the plain-Python loops, only the static types, the punctuation, and std::; the loops themselves are identical. Switch tabs, and copy any of them:

def clamp01(im):
    out = np.empty_like(im)
    H, W, C = im.shape
    for c in range(C):
        for y in range(H):
            for x in range(W):
                out[y, x, c] = min(1.0, max(0.0, im[y, x, c]))
    return out

Image clamp01(const Image &;im) {
    Image out(im.width(), im.height(), im.channels());
    for (int c = 0; c <; im.channels(); c++)
      for (int y = 0; y <; im.height(); y++)
        for (int x = 0; x <; im.width(); x++)
          out(x, y, c) = std::clamp(im(x, y, c), 0.0f, 1.0f);
    return out;
}

def clamp01(im):
    return np.clip(im, 0.0, 1.0)

def luminance(im):
    H, W, C = im.shape
    out = np.empty((H, W, 1))
    for y in range(H):
        for x in range(W):
            out[y, x, 0] = (0.299 * im[y, x, 0]
                          + 0.587 * im[y, x, 1]
                          + 0.114 * im[y, x, 2])
    return out

Image luminance(const Image &;im) {
    Image out(im.width(), im.height(), 1);          // one output channel
    for (int y = 0; y <; im.height(); y++)
      for (int x = 0; x <; im.width(); x++)
        out(x, y, 0) = 0.299f * im(x, y, 0)
                     + 0.587f * im(x, y, 1)
                     + 0.114f * im(x, y, 2);
    return out;
}

def luminance(im):
    return (0.299 * im[:, :, 0:1] + 0.587 * im[:, :, 1:2]
            + 0.114 * im[:, :, 2:3])

def blend(a, b, t):
    out = np.empty_like(a)
    H, W, C = a.shape
    for c in range(C):
        for y in range(H):
            for x in range(W):
                out[y, x, c] = (1.0 - t) * a[y, x, c] + t * b[y, x, c]
    return out

Image blend(const Image &;a, const Image &;b, float t) {
    Image out(a.width(), a.height(), a.channels());
    for (int c = 0; c <; a.channels(); c++)
      for (int y = 0; y <; a.height(); y++)
        for (int x = 0; x <; a.width(); x++)
          out(x, y, c) = (1.0f - t) * a(x, y, c) + t * b(x, y, c);
    return out;
}

def blend(a, b, t):
    return (1.0 - t) * a + t * b

def mean(im):
    H, W, C = im.shape
    total = 0.0
    for c in range(C):
        for y in range(H):
            for x in range(W):
                total += im[y, x, c]
    return total / (H * W * C)

float mean(const Image &;im) {
    float total = 0.0f;
    for (int c = 0; c <; im.channels(); c++)
      for (int y = 0; y <; im.height(); y++)
        for (int x = 0; x <; im.width(); x++)
          total += im(x, y, c);
    return total / (im.width() * im.height() * im.channels());
}

def mean(im):
    return im.mean()

The luminance example retains the course's grayscale weights and function name: on encoded RGB these weights compute a luma-like quantity, not physical linear-light luminance. Inputs are nonempty; RGB examples need three channels, and blending requires matching shapes.

Across all of them the same shapes recur in the C++: const Image & for every input, im.width()/height()/channels() as the loop bounds, float literals with the f suffix, and a fresh Image out(...) to fill and return (or, for a reduction, a plain float). The plain-Python column makes the real point: the C++ loop is not exotic, it is the loop you would write in Python the moment you step off NumPy's vectorized path, with static types and braces bolted on. Once those are muscle memory the translation is mechanical, and making the loop fast is the compiler's job.

Files and compilation🔗⧉

A C++ project is split in two. The header Image.h declares the class and the function signatures; the implementation Image.cpp defines them. To use another module you #include "Image.h", which literally pastes its declarations in. Because a header can be included along several paths, each one is wrapped in an include guard, an #ifndef / #define / #endif sandwich that stops it being pasted twice; it looks archaic, and the template files already have it. A build tool (make, or CMake) reads a build file listing the sources and runs the compiler, so you compile the whole project with one command.

Two syntax traps trip people early. In a .cpp file the scope operator :: says which class a definition belongs to, as in float Image::operator()(int x, int y, int c), because one file can implement several classes. And a class definition needs a semicolon after its closing brace; forget it and the error surfaces, confusingly, in the next file.

Value, reference, and const🔗⧉

This is the shift that trips Python programmers most, because Python hides it. How you declare a function's parameter decides whether it sees a copy or the original, and whether it may change it.

• f(Image im) passes by value: an lvalue argument is copied into the parameter unless the class defines other behavior. Moving a temporary can avoid a copy.
• f(Image &im) passes by reference: the function works on the original, and changes it makes persist. No copy, so it is fast.
• f(Image *im) passes a pointer (a raw memory address). We avoid pointers in this book.
• f(const Image &im) passes by reference but read-only: fast, no copy, and the compiler forbids the function from modifying it. This const & is how you pass most inputs.

A trailing const on a method, int width() const;, is the same promise from the other side: the method will not mutate the object, so getters are marked const and can be called on read-only inputs.

Memory, briefly🔗⧉

An ordinary local object has automatic storage duration and is destroyed when its scope ends; its owned pixel buffer may live in dynamically allocated storage; Image im(w, h, c); is all you need, with no new. Heap allocation with new must be matched by delete, which is where leaks come from, so we lean on std::vector, which owns and frees its own storage. The one sharp edge to remember: array access is unchecked, so v[i] with i out of range does not raise an exception. It silently reads or corrupts unrelated memory, or crashes with a segmentation fault, often far from the actual mistake.

Images in C++🔗⧉

The course wraps an image in a class holding a flat std::vector<float> (often normalized to [0, 1], but intermediate values can lie outside that range) together with its width, height, and channel count. A vector has a single index, so a 3-D image is packed into 1-D by strides. The book, like Halide, stores it plane by plane (all of one channel, then the next):

index of (x, y, c) = c * width * height + y * width + x

An accessor im(x, y, c) hides that arithmetic. It takes x first, unlike NumPy's usual [y, x, c] indexing, and keeps the code consistent with Halide later. The same layout and the reasoning behind it are in Chapter 8.1, Image representation.

Debugging, and the parts that hurt🔗⧉

The recurring frustration of C++ is that errors surface far from their cause: the missing class semicolon breaks the next file, an out-of-bounds write crashes somewhere unrelated, a segmentation fault points at the symptom, not the bug. A few habits from the course make it bearable.

• Read compiler errors from the first one down; later ones are often fallout.
• Test each piece as you write it, on tiny inputs (a 3×3 image, a constant, a single edge) whose answer you can work out by hand. Do not write the whole thing and then run it.
• Print and save intermediates: write images to disk and look at them.
• On a segmentation fault, suspect an array index. Print the index and the size, or assert(0 <= i && i < size).
• Compiler errors and syntax are exactly what search and AI assistants handle well; use them, and acknowledge any AI help per the course policy.

Where to go next🔗⧉

The first problem set (Chapter 45.3, Problem Set 0 — Environment and C++ basics) is a warm-up by design: set up the environment, compile the template, write a few lines with std::vector, and do a brightness or contrast point operation. The psets after it build the Image class, point and neighborhood operations, and onward. For a fuller Python-to-C++ walkthrough, the course points to Goldwasser and Letscher's Python to C++ guide and the cpp4py tutorial.