45.20 The course tutor: a local, book-grounded AI teaching assistant⧉
This book ships with an AI tutor grounded in the same course material. The course tutor is a chatbot whose answers are grounded in this book (together with the lecture slides and the problem-set handouts) and whose job is to help a student understand the material and make progress on the psets without doing the psets for them. It runs, by default, on the student's own laptop. This appendix explains how it is built and why, in the same plain way the rest of the book explains a pipeline.
45.20.1 What it is, and what it is for⧉
The tutor answers two kinds of question differently. A conceptual question (what is a bilateral filter, why does a Gaussian blur smear edges, what does white balance actually do), it answers directly and fully, building intuition first and reaching for the math when the math helps, and it points the student at the section of the book that covers it. A problem-set question (how do I implement this, is my code right, just give me the formula), it treats as a teaching moment: it names the underlying concept, points to the section to read, and offers the next step or a guiding question, then waits for the student to try. Its instructions discourage giving complete solutions, and answer keys are excluded from retrieval; these measures do not guarantee that the model cannot derive or disclose an answer. The tutor is meant to guide students, not provide answers.
This "guide, don't solve" stance is a deliberate echo of the systems that came before it: Harvard's CS50 Duck, which leads students toward answers with the kind of rhetorical question a section leader would ask; Khan Academy's Khanmigo, which redirects a student who tries to extract an answer back to the reasoning; and Georgia Tech's Jill Watson, which grounds every reply in actual course materials and cites them. The tutor borrows from all three.
45.20.2 Local-first⧉
The default tutor runs an open-weights model on the student's own machine: no API key, no per-token cost, with inference performed on the laptop; separately configured log uploads and online fallback can transmit questions. An install-time probe looks at the hardware (memory, and whether there is a capable GPU or Apple-Silicon chip) and picks a model size that will run comfortably: a smaller model on sufficiently capable laptops, a larger and more capable one where there is a GPU. The local-first choice is partly practical (cost and offline use) and partly principled. Local inference avoids sending a question to a remote inference service. Whether it later leaves the laptop through logging is a separate policy and configuration choice.
For a student whose machine cannot run a model locally, there is an optional online backup: the same tutor, served from an instructor-run endpoint, so no one is locked out by hardware. The student never holds a key in either case; the backup keeps any credentials on the server.
45.20.3 Grounded in the book: retrieval-augmented generation⧉
A language model working from memory can confabulate, for example by inventing a citation or misstating a formula. The tutor is built to lean on the book instead. Before it answers, the question is used to retrieve the most relevant passages from a local index of the book's drafts, outline, companion files, slide digests, and the pset handouts; those passages are handed to the model as the context it must answer from. This is retrieval-augmented generation (RAG), and it is the same discipline the authoring pipeline follows: write from the sources, not from memory. The index is built once by the instructor and shipped with the tutor. The next three sections are the details of how that retrieval works, which took a good deal more than similarity search to get right.
Two consequences follow. First, solutions and answer keys are deliberately never put into the index: the tutor cannot retrieve and parrot a worked answer it was never given, which removes one direct leakage route while leaving the model capable of solving some problems from its own training or retrieved explanations. Second, when retrieval comes back weak (the question is only loosely related to anything in the course), the tutor is told to say so ("the course material doesn't really cover this") rather than confidently make something up. This "decline when uncovered" behavior, borrowed from Jill Watson, is the single most useful guard against a tutor that sounds sure and is wrong.
45.20.4 How the retrieval works⧉
The textbook version of RAG is three steps: cut the text into chunks, embed the chunks, hand the model the five nearest to the question. That version works on a product FAQ and fails on a book like this one, for a reason the book creates itself. A thorough text teaches a concept many times over: convolution appears in the Fourier chapter, in the filtering chapter, in demosaicking, and in a pset handout, and a question about it sits close to all of them. The search returns five plausible sections and the one that actually answers is seventh. What follows is what moved this retrieval from plausible to right, with the measurement that decided each choice, because several of them reverse the standard advice. The following experiments describe the book assistant's retrieval system. The standalone tutor has a separate local index and retriever; the repository version reviewed here uses local embedding retrieval and does not automatically inherit every book-assistant feature or result.
The unit is a section: 1,748 of them. A chapter is too coarse to cite usefully and a paragraph is too small to answer from. The first index was parsed from the book's outline rather than its drafts, which had two consequences: the assistant answered from outline prose while the reader was looking at the finished page, and 14 percent of the anchors it cited did not exist on those pages. It is built from the drafts now, and a section's id is its anchor on the page by construction, which reduces anchor mismatches when the index and published pages are synchronized.
Drafted sections run long (median 2,600 characters), so each is cut into 850-character windows spaced 700 apart. A window is embedded together with its chapter title, its section title, and the one-line paraphrase of the section that the analysis pass wrote for it. That paraphrase is worth five points of rank-one accuracy on its own, because it says in a reader's vocabulary what the passage says in the author's. The passage itself is stored next to the vector rather than only pointed at. Only the best-matching chunk of a section is kept, so one long section cannot fill the result list.
Embeddings blur exactly what a student is most likely to type: a name. "Naka-Rushton", "Snell", Figure 2.1.47 have almost no distributional neighborhood, and cosine similarity places them near anything on the same general topic. The same chunks therefore carry an ordinary BM25 word index, and the two rankings are fused by reciprocal rank, each hit contributing 1/(60 + its rank) to its section. BM25 alone matched the embedding at rank one; fusing the two lifted right-section-first from 34 to 42 percent, the largest single gain in the system.
A field name is a different kind of query. "Computer vision" or "optics" is not a question, and no passage is about it more than the others are. Those expand through a topic map instead: every section is tagged with the topics it teaches and the topics it merely applies, and a named field pulls in the sections that teach it, weighted by how central the topic is to each. The catch is that this only helps when the name is most of the query. Ungated, the expansion fired on 117 of our 200 test questions, and a fifteen-word question that happened to contain "depth of field" got a candidate pool flooded with topic-mates. It now requires the matched name to cover at least half the query. A second mechanism repeated the same lesson: a set of logistic probes reads the probability of each topic straight off the query's embedding, and answers "how does the eye adapt to a dark room" with human vision at 0.93 without any of those words appearing. Ungated, the probes dropped right-first from 51 to 35 percent. Restricted to short name-like queries, where the words give the text search nothing to work with, they cost nothing.
Then there is the stage we removed. Standard practice is two rounds: a fast embedding proposes a few dozen candidates, and a cross-encoder that reads question and passage together re-sorts them. It was the biggest win of the first audit, ten points at rank one, once we fed it passages instead of titles. After the other fixes landed it became a liability: on the current index the fused order puts the right section first 48 percent of the time and the same reranker takes that to 32. Nothing about the reranker changed. This reranker helped the earlier candidate set but hurt the improved one. That result is specific to the model, passages, and evaluation; it does not imply that small cross-encoders are always weak. It stays behind a flag, and the lesson generalizes past this system: re-measure every stage after improving any other one.
Retrieval is also a loop rather than a step. The model is given tools and up to seven turns: search sections, pull a whole section, list a chapter's contents, search figures by caption, look at a figure's image, find an interactive demo. It can search, learn the book's vocabulary for the thing it was asked about, search again with better words, then read the section in full before answering. A first search that only lands in the right neighborhood is recoverable that way; a single-shot pipeline has to be right immediately.
45.20.5 Indexing other people's books⧉
Roughly half the index is not this book. Ours is a course in a field that borrows, and a student asking about microfacet BRDFs or std::span is better served by the standard reference than by our summary of it. Outside material is indexed at the depth its license allows, in two tiers. Where the license permits ingestion or the authors have said yes (PBRT, the Freeman, Isola and Torralba vision book, Wandell's Foundations of Vision, the Halide tutorials, the C++ Core Guidelines, the colour-science documentation), the prose is chunked and embedded like ours. Where the material is free to read but not licensed to us (cppreference, Lindbloom's color formulas, Poynton's gamma FAQ, Webvision), only the table of contents is indexed: section titles and URLs, not one word of the text.
In both tiers the text is embedded and then deleted. What persists is a title, a URL, and a vector, which are not verbatim stored passages; embeddings can nevertheless leak information, so they should not be treated as a guarantee of non-recoverability. The assistant sends the reader to the source rather than reproducing it, and no excerpt of someone else's book is stored in ours.
An outside link has to clear two thresholds, and it needs both. The first is a similarity floor. The second is the cross-encoder from the previous section, used here as a yes-or-no gate rather than a ranker, which is the one job it is good at. Before we added it, a question about C++ syntax came back with a link to PBRT's chapter on triangle meshes: prose about code, close enough in embedding space to clear the floor, and useless. Real hits score between 0.016 and 0.48 on the gate. That mesh chapter scored 0.0002, which is also why the veto threshold is 0.005 and not zero: at three decimals the offending score printed as 0.000 and the first version of the check let it through.
45.20.6 Measuring it, and keeping it from going stale⧉
Every number above comes from a fixed set of 200 questions, and building that set was itself a lesson. The obvious evaluation is to query the index with each section's stored paraphrase and check that the section comes back. It scores near 100 percent and means nothing: that paraphrase was written from the section and embedded into its own vector. The questions are instead written from sections the retrieval never sees at query time, phrased the way a student would ask, and we report the rate at which the right section comes back first, in the top five, and in the top ten, plus mean reciprocal rank. Across the fixes described here those went from 32 / 62 / 65 percent to 51 / 81 / 91, and MRR from 0.44 to 0.64. Rebuilding the index shifts any of them by a couple of points with no change in code, so small differences need repeated or paired evaluation before being treated as improvements; four points is an empirical caution here, not a universal significance threshold, which is worth knowing before you celebrate one.
Fifty-one percent right-first sounds low for a system anyone would ship. It is the correct number for a book that teaches convolution in four places: several of the misses are a section that would have answered the question perfectly well, and the answer stage reads the top few and can go fetch more. Recall at five and ten is what governs whether the answer is grounded, which is why those are the numbers we tuned.
The last piece is not retrieval at all, it is bookkeeping. A stale index fails silently: pages change, vectors do not, and the assistant keeps citing a section that has been renumbered or deleted, in exactly the same confident voice it uses when it is right. Reorganizations can change page routes and derived anchors even when stable section identifiers are retained, and this book has been reorganized repeatedly. Publishing therefore refuses to proceed until three checks pass: that the catalog, the stored chapters, the vector index and the built pages agree section by section (drift is re-embedded on the spot), that the retrieval evaluation still scores what it scored last time, and that the links the assistant is able to hand out resolve to pages that exist. The retrieval quality was a few weeks of experiments. The freshness is permanent maintenance, and it is the half that decides whether a student is told something true.
45.20.7 It links, it shows equations, it shows figures⧉
Because the tutor knows which section each answer came from, it cites the book by name and links to it, so a student learns to navigate the text rather than treating the chatbot as a replacement for it. In the browser interface those citations are live links; mathematics is rendered properly (a real fraction, a real summation, not raw LaTeX); and the figures that live in the cited sections are shown alongside the answer. For a subject as visual and as mathematical as this one, that matters: a good figure of the bilateral filter collapsing its weight at an edge explains more than a paragraph, and a rendered equation is far easier to read than its source. A vision-capable model also lets a student upload an image (their own result, an artifact they don't understand) and ask about it directly.
45.20.8 Two front-ends, one core⧉
The same retrieval-and-reasoning core drives two interfaces. A terminal client lives in the student's development environment, can see the code they are working on, and works over a remote connection: natural for the coding-heavy psets. A browser app trades that for rendered equations, inline figures, image upload, and a model menu: better for understanding the material. Students use whichever fits the moment; both log identically.
45.20.9 What the instructor sees⧉
When collection is configured, interactions (the question, passages retrieved, and tutor reply) are logged and uploaded to the instructor's service, because the main value of the logs is showing where students get stuck. A batch process turns those logs into a dashboard: what kinds of help students seek, usage over the term, who is and isn't engaging, and where the book's retrieval comes up empty (a signal that a topic is under-covered). It also runs a pedagogy audit. An independent model reviews each exchange and flags the ones worth a human's attention: a reply that was wrong or misleading, one that drifted from the sources, or one that gave too much away. The instructor gets a short queue of flagged exchanges rather than a full transcript.
45.20.10 Privacy and disclosure⧉
Interaction logs are student records, and they are treated as such. Their collection and use are disclosed in the syllabus; access is restricted; and before any trace is handed to a third-party model for the pedagogy audit, the student's identity is pseudonymised and automated scrubbing attempts to remove personal details; free text can retain identifying context, with the mapping back to real names kept only on the instructor's machine. The tutor presents itself plainly as an AI that can be wrong, and its whole design points the student back into the book rather than standing in for it. It is a study aid built by the same hands, and on the same principles, as the book it teaches.