💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.
jump to

22.17 The course tutor: a local, book-grounded AI teaching assistant⧉

There is a pleasing recursion in a book about computational imaging shipping with a tutor that is, itself, a small piece of the machinery the later parts describe. The course tutor is a chatbot whose knowledge is this book — together with the lecture slides and the problem-set handouts — and whose job is to help a student understand the material and make progress on the psets without doing the psets for them. It runs, by default, on the student's own laptop. This appendix explains how it is built and why, in the same plain way the rest of the book explains a pipeline.

22.17.1 What it is, and what it is for⧉

The tutor answers two kinds of question differently. A conceptual question — what is a bilateral filter, why does a Gaussian blur smear edges, what does white balance actually do — it answers directly and fully, building intuition first and reaching for the math when the math helps, and it points the student at the section of the book that covers it. A problem-set question — how do I implement this, is my code right, just give me the formula — it treats as a teaching moment: it names the underlying concept, points to the section to read, and offers the next step or a guiding question, then waits for the student to try. It will not write the solution or quote an answer key. The model is a patient teaching assistant available at any hour, not an answer service.

This "guide, don't solve" stance is a deliberate echo of the systems that came before it — Harvard's CS50 Duck, which leads students toward answers with the kind of rhetorical question a section leader would ask; Khan Academy's Khanmigo, which redirects a student who tries to extract an answer back to the reasoning; and Georgia Tech's Jill Watson, which grounds every reply in actual course materials and cites them. The tutor borrows from all three.

22.17.2 Local-first⧉

The default tutor runs an open-weights model on the student's own machine — no API key, no per-token cost, and no questions leaving the laptop. An install-time probe looks at the hardware (memory, and whether there is a capable GPU or Apple-Silicon chip) and picks a model size that will run comfortably: a small model on any laptop, a larger and more capable one where there is a GPU. The local-first choice is partly practical (cost and offline use) and partly principled — a student's stumbling questions are private by construction when the model never phones home.

For a student whose machine cannot run a model locally, there is an optional online backup: the same tutor, served from an instructor-run endpoint, so no one is locked out by hardware. The student never holds a key in either case; the backup keeps any credentials on the server.

22.17.3 Grounded in the book: retrieval-augmented generation⧉

A language model left to its own memory will confabulate — invent a plausible-sounding citation, misstate a formula. The tutor is built to lean on the book instead. Before it answers, the question is used to retrieve the most relevant passages from a local index of the book's drafts, outline, companion files, slide digests, and the pset handouts; those passages are handed to the model as the context it must answer from. This is retrieval-augmented generation (RAG), and it is the same discipline the authoring pipeline follows: write from the sources, not from memory. The index is built once by the instructor and shipped with the tutor; the chunks of text are turned into vectors by a small embedding model and searched by similarity at question time.

Two consequences are worth stating. First, solutions and answer keys are deliberately never put into the index — the tutor cannot retrieve and parrot a worked answer it was never given, which keeps the "guide, don't solve" default honest rather than relying on a brittle filter. Second, when retrieval comes back weak — the question is only loosely related to anything in the course — the tutor is told to say so ("the course material doesn't really cover this") rather than confidently make something up. This "decline when uncovered" behaviour, borrowed from Jill Watson, is the single most useful guard against a tutor that sounds sure and is wrong.

Because the tutor knows which section each answer came from, it cites the book by name and links to it, so a student learns to navigate the text rather than treating the chatbot as a replacement for it. In the browser interface those citations are live links; mathematics is rendered properly (a real fraction, a real summation, not raw LaTeX); and the figures that live in the cited sections are shown alongside the answer. For a subject as visual and as mathematical as this one, that matters — a good figure of the bilateral filter collapsing its weight at an edge explains more than a paragraph, and a rendered equation is far easier to read than its source. A vision-capable model also lets a student upload an image — their own result, an artifact they don't understand — and ask about it directly.

22.17.5 Two front-ends, one core⧉

The same retrieval-and-reasoning core drives two interfaces. A terminal client lives in the student's development environment, can see the code they are working on, and works over a remote connection — natural for the coding-heavy psets. A browser app trades that for rendered equations, inline figures, image upload, and a model menu — better for understanding the material. Students use whichever fits the moment; both log identically.

22.17.6 What the instructor sees⧉

Every interaction — the question, the passages retrieved, the tutor's reply — is logged and delivered to the instructor, because knowing where students get stuck is half the value of a tutor. A batch process turns those logs into a dashboard: what kinds of help students seek, usage over the term, who is and isn't engaging, and where the book's retrieval comes up empty (a signal that a topic is under-covered). It also runs a pedagogy audit — an independent model reviews each exchange and flags the ones worth a human's attention: a reply that was wrong or misleading, one that drifted from the sources, or one that gave too much away. The instructor gets a short queue of problems rather than a transcript to wade through.

22.17.7 Privacy and honesty⧉

Interaction logs are student records, and they are treated as such. Their collection and use are disclosed in the syllabus; access is restricted; and before any trace is handed to a third-party model for the pedagogy audit, the student's identity is pseudonymised and personal details are scrubbed, with the mapping back to real names kept only on the instructor's machine. The tutor presents itself plainly as an AI that can be wrong, and its whole design points the student back into the book rather than standing in for it. It is a study aid built by the same hands, and on the same principles, as the book it teaches.