BookNotesSlidesP-sets Syllabusdraft · v0.2.49.1
◀▲⌂▶
💬Comments welcome. To leave a note, select any text and click the note / highlight button that pops up — or open the panel with the tab at the top-right (‹). Notes are visible only inside our private review group.

45.25 The interactive figures — a prompt-by-prompt making-of⧉

A handful of the book's interactive figures are small productions in their own right: a live 2-D wave simulation of diffraction, a full 3-D bilateral grid you can orbit, a real in-browser Lightroom, a from-scratch Beier–Neely morpher, a spherical-and-chromatic-aberration explorer that renders its own point-spread function from a million traced rays. None of them was hand-coded in the usual sense. Each was built and refined by prompting an AI coding agent: describe the idea, inspect the result, and request the next fix. This appendix collects the prompt histories for 25 of these demos, and it reproduces the actual words used, in order.

Across those 25 demos the table records 111 prompts. That count understates the work: some demos took a single sentence to specify, while others (the sampling pipelines, the lens optimizer, the aberration explorer) needed a dozen or more rounds of "not quite, here's what's wrong." Reading the sequences back, a pattern recurs. The first prompt asks for the thing; almost all the rest are physics and rendering corrections (wrong boundary conditions, wrong spectrum colors, wrong ray sampling, wrong focus), the unglamorous back-and-forth that turns a plausible-looking widget into a more faithful demonstration; the prompt history itself is not proof that all errors were caught.

One caveat, stated plainly: these counts are a lower bound. The prompt log is a partial, partly backfilled record. Some of the earliest creation prompts were never captured, and several of the demos here were assembled largely by a separate agent working in the interactive sandbox, whose prompts are not in this log at all, so a few sections rest on only the one or two logged prompts that clearly touch them, and are marked as such. The true iteration count can be higher, sometimes much higher, than the tally shown. Concretely: known prompts are simply missing. The request to add shadow/highlight recovery to the mini-Lightroom, for instance, was given but never logged, so it appears in neither the count nor the section. (Going forward, every session now logs the author's prompts automatically, so later editions of this appendix will be more complete.)

45.25.1 Summary⧉

The figure numbers below record the historical versions discussed in the prompts; use the chapter links and figure IDs to locate the current demonstrations.

DemoHistorical figureHost chapter# prompts
Chapter 2.1, Light and optics\, Figure 2.1.182.1.8Light and optics5
Chapter 2.1, Light and optics\, Figure 2.1.212.1.14Light and optics9
Chapter 2.1, Light and optics\, Figure 2.1.462.1.29Light and optics7
Exposure-triangle simulator, Figure 5.2.62.11.5Exposure1
Chapter 8.1, Image representation\, Figure 8.1.43.1.4Image representation5
Chapter 8.11, File formats and compression\, Figure 8.11.113.17.9File formats and compression4
Chapter 8.11, File formats and compression\, Figure 8.11.123.17.10File formats and compression2
Chapter 6.4, Sampling, downsampling, and aliasing\, Figure 6.4.53.10.4Sampling, downsampling and aliasing13
Chapter 6.4, Sampling, downsampling, and aliasing\, Figure 6.4.73.10.6Sampling, downsampling and aliasing8
Chapter 6.5, Resampling and upsampling\, Figure 6.5.103.11.13Resampling3
Chapter 6.5, Resampling and upsampling\, Figure 6.5.163.11.18Resampling4
Chapter 8.12, Recap ISP, non-destructive editing\, Figure 8.12.43.18.4Recap: ISP and non-destructive editing2
Chapter 10.8, Poisson image editing\, Figure 10.8.55.1.5Poisson image editing1
Chapter 10.1, Bilateral filtering\, Figure 10.1.105.2.15Bilateral filtering7
Chapter 11.6, Perspective distortion and its correction\, Figure 11.6.136.6.13Perspective distortion and its correction2
Beier–Neely morph, Figure 11.3.46.3.4Morphing1
Chapter 12.10, Face tracking\, Figure 12.10.27.10.2Face tracking7
Chapter 22.4, Lens optimization\, Figure 22.4.49.4.4Lens optimization4
Full auto-panorama, Figure 20.3.910.7.9Blending4
Chapter 28.3, Auto curation\, Figure 28.3.211.3.2Auto curation2
Chapter 28.6, Photobios\, Figure 28.6.211.7.2Photobios2
Chapter 31.4, Refocusing and synthetic aperture\, Figure 31.4.213.4.2Refocusing and synthetic aperture4
Chroma key, Figure 19.3.18.11.7Compositing, segmentation and matting2
Portrait-lighting simulator, Figure 5.4.52.11.26Illumination and the flash1
Chapter 22.1, Aberrations and optical challenges\, Figure 22.1.69.1.6Aberrations and optical challenges11
Total25 demos111

45.25.2 Lorentz resonance: Figure 2.1.18 (Figure 2.1.8)⧉

Light–matter interaction as driven bound electrons, from which the index of refraction n(ω) and absorption emerge (Chapter 2.1, Light and optics).

fig-lorentz-resonance
Figure 2.1.18 (reproduced). An incoming wave drives bound electrons that oscillate, resonate, and re-radiate, so that refraction and absorption fall out of a single mechanical picture.

Prompts (5):

  1. When introducing light-matter interaction, discuss how a lot of it (including index of refraction and absorption) relate to the sizeresonance of atoms/molecules.
  2. … New interactive demo for light-matter interaction & resonance.
  3. [Fig 2.1.7 lorentz-resonance] Make the incoming wave clearer (oscillating sine L→R, updated as it passes the resonator); a few resonators; electrons w/ slightly elliptical (vertically stretched) motion?; is my mental model correct?; material presets + explicit light-wavelength control + separate resonance-frequency slider.
  4. yes [rebuild 02-01]. and yes [add the Lissajous ellipse].
  5. … Figure 2.1.7: make the default at resonance; make the resonance & damping sliders update when a material is chosen. …

45.25.3 Rainbow droplet: Figure 2.1.21 (Figure 2.1.14)⧉

Rainbows built by ray-tracing a water drop, with the outgoing-angle caustic that makes the bow (Chapter 2.1, Light and optics).

fig-rainbow-droplet
Figure 2.1.21 (reproduced). Rays enter a spherical water droplet, refract, reflect internally, and exit, piling up at the stationary deviation angle to form the rainbow.

Prompts (9):

  1. With the discussion of rainbows, add my rainbow demo pic which I put in photos-just-for-book with rainbow prefix. Say the glass sphere stands for a water droplet and then discuss. …
  2. Sidebar: rainbows in Light and optics is too short; add my glass-sphere demo photos; explain dispersion alone doesn't explain the rainbow (no rainbow after two refractions); add interactive figures of the droplet ray paths (2D, vary where the ray hits) with outgoing-angle plots.
  3. For the rainbow: Show dispersion by default. Also include the corresponding wavelgths in the 1D plot. Include a second photo of the glass sphere, the one where you see all the rays.
  4. It's rainbow-20251109at14h00m19s-DSC02875 in photos-just-for-book
  5. Rainbow interactive figure: add a checkbox to zoom into the impact-parameter range near the caustic (affecting both slider and plot); remove the index-of-refraction slider (water only); remove the dispersion checkbox (always on); show the integral on the side of the plot. …
  6. Fig 2.1.10 (rainbow-droplet, actually 2.1.11): zoomed ray-density should fall off SHARPLY not linearly; add an RGB color stripe = integral over all wavelengths; fix 'spike rainbow' text overlap; label Alexander's band + bright inside-bow region. …
  7. … Figure 2.1.11. for the zoomed in version, you need to compute the integral more finely for the rightmost display. Right now it's spiky. …
  8. … Fig 2.1.11: even more wavelength samples for the integral. …
  9. For the Rainbow demo, the secondary rainbow paths look wrong. …

45.25.4 Diffraction wave simulation: Figure 2.1.46 (Figure 2.1.29)⧉

A live 2-D wave simulation of diffraction through an aperture (Chapter 2.1, Light and optics).

fig-diffraction-wave-sim
Figure 2.1.46 (reproduced). A time-stepped 2-D wave passes through a slit and bends around its edges, the wavefronts bowing and spreading into the geometric shadow.

Prompts (7):

  1. … Generate a simulation showing diffraction through an aperture, an actual 2D wave simulation, animated over time; keep the existing diffraction figure but add a new one with the simulation. …
  2. Figure 2.1.23 is flickering like crazy. Slow things down. Also do not let it play by default and make the play/pause checkbox into a big button. …
  3. Figure 2.1.23 is still more chaotic than I'd expect. What are the boundary conditions? Do the waves continue 'off screen' or do they get reflected by the walls?
  4. For the diffraction demo, is it possible to get a bigger domain? Make the figure full width and maybe 50% taller.
  5. What is the boundary condition on the rightmost wall? I feel that the reflection interferes with the diffraction effect.
  6. How about the top, bottom and left boundary? Also I feel that now the simulation is a bit too slow. Try to speed up by a factor of 2.
  7. The demo takes a while to reach steady state when changing lambda. Should we somehow dampen the existing energy? + (central-lobe q) let's keep it honest, just label it, say it's near field/Fresnel.

45.25.5 Exposure-triangle simulator: Figure 5.2.6 (Figure 2.11.5)⧉

A full 3-D simulation of depth of field and motion blur as aperture, shutter, and ISO trade off (Chapter 5.2, Exposure and capture timing).

fig-exposure-triangle-sim
Figure 5.2.6 (reproduced). A 3-D scene rendered as the three exposure controls move together, so the same brightness can be reached with different depth of field and motion blur.

Prompts (1):

  1. … Let's generate 3D models with meshy for our demos / For general use: a few cameras, an eye / … / Generate 3D models of human beings. Make sure to be diverse in gender and race. Give them clothes and all that. / Acknowledge the use of meshy when using such models at the end of captions. …

No prompt in the log is specific to the exposure-triangle / depth-of-field-and-motion-blur sim itself: the sims were built in the interactive sandbox (largely by another instance), and only this batch prompt's general Meshy-subjects clauses clearly apply. Attribution is weak.

45.25.6 Repeated quantization: Figure 8.1.4 (Figure 3.1.4)⧉

Repeatedly darkening an 8-bit image and brightening it back, re-quantizing to 8 bits each round, so the round-trips compound into visible banding that dithering can break up (Chapter 8.1, Image representation).

fig-repeated-quantization
Figure 8.1.4 (reproduced). From an 8-bit image, the exposure is lowered to a fraction of its value over N steps and raised back, re-quantizing to 8 bits at every step, so the 2N round-trips compound into posterization banding, with an optional dither (pure-random or error-diffusion) that averages the bands away.

Prompts (5):

  1. In 3.1.3, add an interactive demo what happens with repeated quantization. Take input images, gamma encode, quantize to 8 bits. Then repeat N times: Lower exposure by nth root of 2, then raise it N time by the same amount. Each time, do the proper linear exposure correction in float but re-quantize to 8 bits. The middle image should be half as dark. Show the input, the middle image, and the final image. The user can select N with a slider and select the image. Add an option to stochastically dither when re-quantizing. Pure random or error diffusion.
  2. Figure 3.1.4. increase max to 64. Add hovering closeup. Also add a slider to control the number of EVs to darken (from 0 to 5), ie for the intermediate.
  3. Figure 3.1.4. make the darkening go to -7. I am surprised that things look so good.
  4. Figure 3.1.4. There must be a bug. When I do -7EV and back, I see almost no loss... push the EV slider to -12.
  5. Figure 3.1.4. add PSR under the photo (compared to original)

45.25.7 JPEG generation loss: Figure 8.11.11 (Figure 3.17.9)⧉

What happens when you keep opening and re-saving a file as JPEG, each round-trip through the browser's real codec compounds, though it quickly settles to a fixed point (Chapter 8.11, File formats and compression).

fig-jpeg-generation-loss
Figure 8.11.11 (reproduced). An image is saved and re-opened through the browser's real JPEG codec N times at a chosen quality; generation loss compounds at first but quickly settles into a fixed point (coefficient quantization is nearly idempotent), so the degradation is far milder than intuition suggests (the exception being loaders that "prettify" blocks on open).

Prompts (4):

  1. In the JPEG chapter, add a discussion that some JPEG loading code (Photoshop) may try to reduce blocking and DCT artifacts, which can cause weird things if you iterate loading/saving many times.
  2. In 3.17.4 JPEG: compression by perception, let's explore what happens when you keep opening and saving a file as JPG. Users can choose JPEG quality and the number of times you save (up to 10). No need to actually save, but take the image, convert to JPG, convert back to 8 bit values N times.
  3. can we get jpeg quality to go even lower?
  4. Figure 3.17.9. The demo flashes when sliding the number of time saved. Fix it.

45.25.8 Exposure round-trip (JPEG): Figure 8.11.12 (Figure 3.17.10)⧉

What happens when you keep editing and re-saving a JPEG: 8-bit banding and generation loss compounding (Chapter 8.11, File formats and compression).

fig-exposure-jpeg-roundtrip
Figure 8.11.12 (reproduced). An image is repeatedly exposure-corrected, quantized to 8 bits, and re-encoded as a lossy JPEG, so both quantization banding and DCT artifacts accumulate over the iterations.

Prompts (2):

  1. Let's do a combination of Figure 3.17.9. and Figure 3.1.4.. We'll do the same as for Figure 3.1.4. (iterated exposure correction with 8 bit quantization) but in addition, at each iteration, we will convert to JPG of a given quality and do the lossy DCT quantization. Place this new figure after Figure 3.17.9.
  2. …Figure 3.17.10. the text under the three images before the controls is too wide. Wrap around.

45.25.9 1-D sampling pipeline: Figure 6.4.5 (Figure 3.10.4)⧉

1-D Fourier sampling and aliasing: signal → prefilter → sample → reconstruct, with continuous spectra and replicas (Chapter 6.4, Sampling, downsampling, and aliasing).

fig-sampling-1d-demo
Figure 6.4.5 (reproduced). A 1-D signal is prefiltered, sampled, and reconstructed while the panel shows the primal signal and its Fourier spectrum with the sampling replicas at each stage.

Prompts (13):

  1. …Move 3.7.9 before 3.7.8. Promote 3.7.8 to a chapter (Down)Sampling and aliasing; add 1D and 2D sampling/prefiltering demos (prudential). …
  2. …Figure 3.7.17: why pulldown in interactive? add tick marks for sample locations; red sine is upside down, fix + make fainter; allow control of sampling grid. …
  3. …Figure 3.7.19: include both spatial and Fourier; red part only starts mid first lobe; don't show prefilter yet. …
  4. Figure 3.17.4. let's make the spectra more continuous and change the input functions. We can use the same functions but progressively vary the period from left to right. The slider controls the middle cycle. Then pilot spectra as continuous functions, and plot the replica in a different color plus the sum of original and replicas.
  5. …Figure 3.18.5: try different signals: shaped white noise, shaped in Fourier with 1) 1/f 2) zero-mean gaussian 3) gaussian centered on a high frequency 4) an image scanline. …
  6. Figure 3.10.5. make gaussian low pass the default. Are you sure the various spectra have the right color in the rightmost plot? Let's add a fourth display after reconstruction filter. Let people choose reconstruction including box, Gaussian, bicubic. Box is default.
  7. …Figure 3.10.5. the coloring of the spectra must be wrong because even when I select a perfect pre filter I can't see the primary spectrum. …
  8. …Figure 3.10.5. The Fourier domain plots go too far. Reduce the max frequency by a factor of 2, Also add a single sine wave as one of the signals.
  9. Figure 3.10.5. The Fourier scale should go to 2.3x Nyquist. …
  10. …Figure 3.10.5. "sampled" should just be a discrete set of spikes. show the pre filtered as a dashed line. For the reconstructed primal, de-emphaseize the samples. Pre-filtered spectrum should be scaled by the pre filter's spectrum! Sampled spectrum: The color of the spectra are still wrong. Teh original should be shaded gray not reddish. Show the spectrum of the reconstruction filter. Include a perfect reconstruction as an option. Make the single sine the default. …
  11. …Make Figure 3.10.5.'s Fourier axis go to 3.3x Nyquist. By default make the reconstruction a box.
  12. Figure 3.10.5. turn single sine -> periodic function. And make the function dropdown select which function (sine, box, etc.) …
  13. …Figure 3.10.5. Let's simplify and only keep the periodic functions. Remove the signal dropdown: only keep Periodic one frequency. Then remove the sliders sweep (chirp) noise shape noise seed …

45.25.10 2-D sampling pipeline: Figure 6.4.7 (Figure 3.10.6)⧉

The 2-D sampling / downsampling / aliasing pipeline, image and Fourier magnitude side by side (Chapter 6.4, Sampling, downsampling, and aliasing).

fig-sampling-2d-pipeline
Figure 6.4.7 (reproduced). A 2-D signal (a curved or straight edge, or an oriented sine) is prefiltered, sampled, and reconstructed, with each stage shown as an image and its windowed Fourier magnitude.

Prompts (8):

  1. Then let's do a 2D version of Figure 3.17.4 with real images or the data from Figure 3.17.5. . Same as Figure 3.17.4, show the 3 steps and both primal and Fourier on two rows.
  2. …Let's do a 2D version of Figure 3.10.5. Same 4 columns and two rows (input, pre filtered, sampled, reconstructed). Input signals will be edges with user-controlled curvature and slopes; or sine waves with user-controlled frequency and orientation. …
  3. Figure 3.10.7. Use windowing for the Fourier transform. Color teh primary vs aliases in teh Fourier domain differently. The sampled image (with the curved edge) looks rotated by 180 degrees. Make sure the image is truly binary for the curved edge. When I apply no pre filter, I should only see black and white sampled values. …
  4. …Figure 3.10.7. Is the processing done in the primal? Let's simplify and remove the curvature for the edge. Crop the images a bit so that there are no edge effects. Show nyquist boundaries as blue dashed lines in pre-filtered. The Fourier domain should extend 3.3x Nyquist.
  5. …Figure 3.10.7. the Fourier view of the edge has a lot of parallel bands. Is it properly windowed? Maybe just replace with the theoretical Fourier transform of a straight edge.
  6. Figure 3.10.7. the straight edge spectrum is completely wrong. It's tiny and not elongated enough. It should be on a straight line orthogonal to the edge.
  7. And why is the original signal all jagged. Render the original signal at full resolution.
  8. Figure 3.10.7. for the edge, the sampled spectrum has discontinuities at horizontal lines that appear to the middle of the original spectrum and all replicas. Finally the reconstructed signal is wrong and does not change when the prefilter or reconstruction filters are changed,

45.25.11 Mitchell–Netravali bicubic (B,C): Figure 6.5.10 (Figure 3.11.13)⧉

The Mitchell–Netravali bicubic (B,C) family explorer, with kernel, frequency response, and test patterns (Chapter 6.5, Resampling and upsampling).

fig-bicubic-bc
Figure 6.5.10 (reproduced). The two Mitchell–Netravali parameters B and C reshape the cubic reconstruction kernel, trading blur against ringing on a unified up/down-sampling zoom of real test images.

Prompts (3):

  1. Figure 3.11.13. the zone plate looks completely wrong. The source is just a ton of little circles all over the place. Also what happened to the other images? There used to be a closeup of the prudential that was working really well. Make upsampling the default.
  2. Figure 3.11.13. make the zone plate and resolution charts thumbnails with the other images. No separate dropdown. Also the scale of both the zone plate and the resolution chart is wrong. They are too zoomed in.
  3. Figure 3.11.13. how are the zone plate and resolution charts handled for upsampling? The nearest neighbor looks too good. You need to render these images at the same resolution as the other inputs (with proper prefiltering like lanczos) and then do the same resampling. Also simplify controls. No need to have different modes for up and downsampling, just a unified zoom slider from 1/4 to 4x. …

45.25.12 Rotation resample challenge: Figure 6.5.16 (Figure 3.11.18)⧉

What happens if you rotate and resample an image too many times (Chapter 6.5, Resampling and upsampling).

fig-rotation-resample-interactive
Figure 6.5.16 (reproduced). An image is rotated in many small steps, each a full resample, so interpolation blur compounds and the iterated reconstruction filter eats the high frequencies.

Prompts (4):

  1. Make another interactive version of Figure 3.11.16. (but keep the static as a separate figure). Just show one image. The user has sliders to control the total angle rotated and the angle of individual rotation (where the total angle is rounded to the nearest multiple of individual rotation). Then a dropdown to choose the resampling filter (nearest, box, bilinear, lanczos, bicubic), and maybe our usual Mitchell 2D interface to choose the bicubic.
  2. Figure 3.11.17. use the same iguana as for Figure 3.11.16. / For the step (each rotation, use a log progression and snap to perfect divisors of 360
  3. Figure 3.11.17. Show the Fourier spectrum of the iterated reconstruction filter. More precisely Show an approximate version of the reconstruction filter iterated as many times as we have performed rotations. Make it simple, just the 1D spectrum exponentiated (but note that in order [msg cut off]). …
  4. Figure 3.11.16. should go before 3.11.11 Prefiltering when the transform isn't a clean scale …

45.25.13 Mini-Lightroom: Figure 8.12.4 (Figure 3.18.4)⧉

A mini in-browser Lightroom: a real per-pixel raw-develop pipeline driven by a Lightroom-style panel (Chapter 8.12, Recap ISP, non-destructive editing).

fig-mini-lightroom
Figure 8.12.4 (reproduced). A raw image is developed live through demosaic, denoise, exposure, contrast, black/white point, curves, white balance, saturation, vibrance, sharpening, highlight rolloff, and vignette, all from a slider panel.

Prompts (2):

  1. …3.15 mini-Lightroom demo (demosaic, denoise, exposure, contrast, black/white point, curves, white balance, beauty curve, saturation, vibrance, sharpening, highlight rolloff, JPG, vignette) + Lightroom-style slider UI + show editing metadata (yaml). …
  2. …Where is my Interactive: a mini-Lightroom??? Why is it not working? …

45.25.14 Poisson blending: Figure 10.8.5 (Figure 5.1.5)⧉

Poisson image editing, with linear, gamma, and log spaces compared (Chapter 10.8, Poisson image editing).

fig-poisson-blend
Figure 10.8.5 (reproduced). A source patch is seamlessly cloned into a target by solving a Poisson equation on its gradients, compared across linear, gamma, and Chong-log working spaces with optional mixed gradients.

Prompts (1):

  1. …Fix the poisson demo [beauty pass only after blending; Poisson in linear/gamma/Chong]. …

45.25.15 Bilateral grid (3-D): Figure 10.1.10 (Figure 5.2.15)⧉

The 3-D bilateral grid (splat, blur, slice) rendered as an orbitable volume (Chapter 10.1, Bilateral filtering).

fig-bilateral-grid-3d
Figure 10.1.10 (reproduced). An image is splatted into a 3-D grid of space and intensity, blurred with a 3-D Gaussian, and sliced back, the voxels drawn back-to-front with opacity proportional to their accumulated weight.

Prompts (7):

  1. Let's build a 3D interactive demo of the bilateral grid. Given an input image, let's visualize the grid with a level of transparency proportional to the sum of its weights. Make sure you properly render the grid back to front for proper transparency handling. Then the user can choose to display the original grid or the 3D-Gaussian-blurred version, and whether to overlay the input image as a height field or not. To display the height field image, make sure it too is properly sorted back to front and interleaved with the grid. Convert input images to grayscale first. Next to the grid, show the image or the output. When hovering on the image, show the corresponding 3D point.
  2. Figure 5.2.15. has a big black space on the left and as a result the figure goes in the margin on the right. Also I feel the voxels are rendered too small and don't occupy their full cube.
  3. Love it. Now add real images from our hdr data. In particular vine, design center, antelope. DO the bilateral; on the log luma. Add a tiny epsilon to values to prevent logs from going crazy (like add half the smallest non-zero value)
  4. Are the voxels actually rendered as full cubes ? I feel they are just splatted as squares?
  5. The original image is hard to see as height field. Render it as a meshed height field (quads or triangles for each pixel) with partial transparency.
  6. Maybe the height field is too much. Try to only render it as wireframe. And make sure the original image renders properly from all directions.
  7. Awesome. For some reason, the height field has these almost vertical walls at the boundary. Remove these. Maybe you read 0 values outside the image? Only on the left and right as far as I can see. Or maybe it's just a problem with the vine image and all you need to do is crop that one by a couple pixels to the side?

45.25.16 Perspective montage: Figure 11.6.13 (Figure 6.6.13)⧉

The holiday-card multi-perspective montage, with 3-D models made for the demo via Meshy (from Fredo's former student Yuanming Hu) (Chapter 11.6, Perspective distortion and its correction).

fig-perspective-montage
Figure 11.6.13 (reproduced). A grid of boxes, each shot head-on with its own perspective, is set against the same boxes rendered as one scene seen from a single camera, with a focal-length slider driving both.

Prompts (2):

  1. I want to build a 3D demo to discuss the perspective in the montage I made for a family new year card (see attachment). This should go in the perspective distortion in 6.7. See and cite https://www.thecomputationalphotographer.net/2023/01/family-in-a-box/ for how I made it. I want the demo to show the two conditions: The montage where each box has its own perspective and the implied scene where there is a single scene with 4x13 boxes side by side seen from a single camera. The demo will be in real 3D. Generate the box yourself. Make an educated guess for the size of the box. Then use meshy to generate 3D models for the people, cats and stuffies inside the boxes. It doesn't need to be a perfect match but try. Then assemble the full merged 3D scene on the one hand with all the boxes and their content in the same space and the 2D composite of the 4x13 individual perspectives centered on each box. The user should be able to select the distance or focal length. Both the unified 3D view and the composite 4x13 montage should be shown on top of each other in the demo. Cite both my blog post and this article https://jov.arvojournals.org/article.aspx?articleid=2793609. Make sure you also discuss https://pmc.ncbi.nlm.nih.gov/articles/PMC3440305/ regarding perspective distortion.
  2. …Figure 6.7.6. the math on the per-box focal length must be wrong. Division instead of multiplication? It should be more telephoto. …

45.25.17 Beier–Neely morph: Figure 11.3.4 (Figure 6.3.4)⧉

Image morphing by the Beier–Neely feature-line method (Chapter 11.3, Morphing).

fig-beier-neely-demo
Figure 11.3.4 (reproduced). Two faces are morphed by drawing corresponding feature line segments, warping each image toward the other, and cross-dissolving as a time slider sweeps from one to the other.

Prompts (1):

  1. For the morphing chapter, implement a full version of Beier and Neely in javascript so that we can create interactive figures. People should be able to enter their before/after segments and see the morph. Use my photo and the werewolf as default. Show the individual warps and the blend. Use inspiration from the slides. People should be able to control the time t or play the animation. Then remake figure 6.3.2. with real images. …

45.25.18 Live face landmarks: Figure 12.10.2 (Figure 7.10.2)⧉

Live face tracking with makeup and silly hats via the device camera (Chapter 12.10, Face tracking).

fig-face-landmarks-live
Figure 12.10.2 (reproduced). A dense face mesh tracks your face through the webcam in real time, letting you paint on makeup or pin a 3-D hat that follows your head as it turns and tilts.

Prompts (7):

  1. …When talking about face landmarks in the matching/tracking part have a real time demo. Include the ability to draw makeup or add hats or replace with 3d meshes. …
  2. …Figure 7.9.2. make it bigger. Ask meshy to generate 3D hats. The start camera button should become stop once the camera is running. …
  3. go with the hats
  4. The hats are not well centered. They are in front of the person.
  5. The hat demo is still too much in front of the person and a bit too high now. Although not by much.
  6. …Figure 7.9.2 (hats): right height now but too far forward, centered on forehead rather than center of head; also when I stop the camera the hat stays displayed on top of the controls. …
  7. For the hat demo, I feel there is a coordinate system problem. The hat doesn't seem to tilt properly when my head tilts up and down. Meanwhile I got the best results with -0.25, 0.38, 0.23 …

45.25.19 Lens optimizer: Figure 22.4.4 (Figure 9.4.4)⧉

Interactive lens optimization: a multi-element lens optimized by damped least squares over fields and wavelengths (Chapter 22.4, Lens optimization).

fig-lens-optimizer-demo
Figure 22.4.4 (reproduced). A multi-element lens is optimized live, its surfaces flexing as damped least squares drives down a merit function while spot diagrams and the merit curve update.

Prompts (4):

  1. … in lens optimization, talk about main software and how people use them in practice …
  2. … Figure 9.3.4: rms improvement not very significant; add slider to access different times in the optimization and see spot diagrams evolve; button to reset the optimization but not the design; what does the horizontal axis of the spot diagrams represent?; increase max number of rays; include a zoomed in version of the rays near each image point (next to the corresponding spot diagram); try a 3D version of Figure 9.3.4 (full-fledged 3D lens and rays, spot diagrams fully 2D).
  3. … The lens optimizer is getting busy. Let's make the controls for the individual elements collapsable and collapsed by default. | What is the sensor size? Is it full-frame? | I am getting disappointing results for wide-angle lenses. | I am still getting some interpenetration. | I get situations where the final RMS is worse than the initial one. Why don't I at least get the minimum value of merit ever reached? | The merit plot re-scales weirdly (super narrow). Is there a bug in the rescaling? | The RMS I see displayed is not consistent with the one shown at the bottom. Use the same rays for display and optimization. …
  4. … [Lens optimizer] it looks pretty wrong right now. The displayed merit curve looks mostly flat and doesn't match the spot diagrams. The final merit is wayyy worse than the initialization.

45.25.20 Full auto-panorama: Figure 20.3.9 (Figure 10.7.9)⧉

Full panorama stitching, the whole automatic pipeline from images to a blended mosaic (Chapter 20.3, Blending).

fig-pano-stitch
Figure 20.3.9 (reproduced). Several overlapping photos are matched, aligned by homographies, reprojected to a common surface, and multiband-blended into a single seamless panorama.

Prompts (4):

  1. …Move full end-to-end pano demo to after blending (end 10.6.3) + forward ref at end 10.5.3; move Figure 10.5.8 to end 10.6.3. …
  2. …Figure 10.4.7 → interactive demo (3 images in photos-just-for-book). …
  3. …Figure 10.5.7: cylindrical/spherical warp should be final reprojection not pre-warp; match links hard to see (bigger lines, two offset colors); include Boston skyline + guedelon; show weights and high/low frequencies in inputs; precompute all 3 blends to switch quickly. …
  4. …Make the final big pano demo usable with computer/phone camera.

45.25.21 CLIP-IQA curation: Figure 28.3.2 (Figure 11.3.2)⧉

Automatic photo rating and curation with CLIP-IQA, scoring a set of images in-browser (Chapter 28.3, Auto curation).

fig-clip-iqa
Figure 28.3.2 (reproduced). A set of uploaded photos is scored and ranked by a real CLIP model probed with quality prompts, all running client-side.

Prompts (2):

  1. …Create a demo of CLIP-IQA for chapter 11.3. People should be able to upload a set of images and get scores. …
  2. go

45.25.22 Photobio time-lapse: Figure 28.6.2 (Figure 11.7.2)⧉

Photobios: aligned portraits picked and cross-faded into an aging time-lapse (Chapter 28.6, Photobios).

fig-photobio-demo
Figure 28.6.2 (reproduced). A collection of a person's portraits is aligned and sequenced so that a smooth cross-faded time-lapse plays across years of their face.

Prompts (2):

  1. …videos-from-fredo: 3 videos for 11.7, include all 3. …
  2. …Replace Figure 11.7.1 with frames from the video results.

45.25.23 Refocus shift geometry: Figure 31.4.2 (Figure 13.4.2)⧉

Light-field refocusing by shift-and-add, with the optical geometry and a real captured light field (Chapter 31.4, Refocusing and synthetic aperture).

fig-refocus-shift-geometry
Figure 31.4.2 (reproduced). The optical configuration for refocusing is drawn alongside a real captured light field that refocuses as one slider shifts and adds the sub-aperture views.

Prompts (4):

  1. Outline the light field chapters based on my slides, transcripts, and appropriate papers. Plan figures. Create them. Draft chapters.
  2. Make Figure 13.4.2. interactive. People can choose the depth of the virtual plane. Plus make an actual real demo
  3. Figure 13.4.2 crazy for most of range, inconsistent with 13.4.5, use chess scene.
  4. Figure 13.4.2. the range of focusing distances for the chess light field offers mostly blurry images. I would say that only slopes between -1.2 and 0.2 are useful. I would restrict the range of refocusing for the real light field (chess) but keep the 2D optical demo range the same. The figure still says: actual demo · Lytro Illum IMG_0228 but it's not that data.

45.25.24 Chroma key: Figure 19.3.1 (Figure 8.11.7)⧉

Green-screen matting / chroma key, with a color-space view, camera-capable (Compositing, segmentation and matting).

fig-chroma-key
Figure 19.3.1 (reproduced). A green or blue background is keyed out of a live or uploaded image, with a color-space plot showing which cluster of colors is being pulled to transparent.

Prompts (2):

  1. …Add a real interactive demo of blue green or chroma key matting. It should be able to use the camera. …
  2. Add color space visualization to green screen matting.

45.25.25 Portrait-lighting simulator: Figure 5.4.5 (Figure 2.11.26)⧉

A 3-D portrait-lighting simulator: move the key light and shape a portrait (Chapter 5.4, Working with light).

fig-portrait-lighting-sim
Figure 5.4.5 (reproduced). A 3-D head is lit by a movable key light and softbox umbrella, so classic portrait-lighting patterns emerge as the light swings around the face.

Prompts (1):

  1. … Let's generate 3D models with meshy for our demos / … / For the lighting demo, generate fancy lighting umbrellas. / Generate 3D models of human beings. Make sure to be diverse in gender and race. Give them clothes and all that. / Acknowledge the use of meshy when using such models at the end of captions. …

This batch prompt is the only logged prompt that clearly touches the portrait-lighting sim, via its "lighting demo … lighting umbrellas" clause plus the general diverse-Meshy-subjects clauses. The sim's original creation prompts were made in the interactive sandbox and are not in this log.

45.25.26 Aberration explorer: Figure 22.1.6 (Figure 9.1.6)⧉

Spherical, chromatic, and coma aberration explorer with a simulated point-spread function (Chapter 22.1, Aberrations and optical challenges).

fig-aberration-explorer
Figure 22.1.6 (reproduced). Rays from infinity strike a lens at a controllable angle and wavelength, tracing spherical, chromatic, and coma aberrations in 2-D and 3-D while a point-spread function is Monte-Carlo rendered from up to a million rays.

Prompts (11):

  1. Figure 7.1.6. maybe exaggerate the aberration a tad more (maybe a factor of two. Also the fringes on the right are missing the green …
  2. Figure 7.1.6. the ray diagram is wrong in the dynamic versions. Are you using actual ray tracing? The rays through the center of the lens should be roughly straight. …
  3. In chromatic aberrations, show simulations of optical solutions, e.g. doublets or diffractive optics or special glass or more advanced with more elements. Use the ray tracing code we have. 2D is fine. Either show one wavelength all rays coming from a point or one single incident ray but multiple wavelengths. People should control wavelength. Show curve of misfocus as function of wavelength. …
  4. Reuse a subset of the code from Figure 9.4.4 to make a spherical, somatic and chromatic aberration interactive figure for 9.1. The user should control the incoming light angle and be able to display a single or multiple wavelength. They can control the wavelength in the case of single wavelength. For multiple, show just 3 for the simulation but sample finely for the PSF. Show a 3D as well as a 2D diagram, and a simulation of the PSF at the chosen point. Sample enough rays to make the PSF look good (progressive rendering, up to 1000 rays). Sample wavelength and aperture and use stratified sampling. Use rays coming from infinity (parallel).
  5. Figure 9.1.6. use more rays. At least 10x. Zoom in more for the 3D view. Is the sensor placed correctly? I feel it should be a bit closer to the lens for better focus.
  6. Figure 9.1.6. I want more rays for the PSF simulation. Have a slider for the max number of rays. Make it a log progression from 10^3 to 10^6. Also make f/1.4 the default. Note that the optical diagrams can and should have fewer rays than for the PSF computation.
  7. Figure 9.1.6. make 1M rays the default and remove slider.
  8. For 3D, put the center of rotation for the UI near the focus point. When changing the incidence angle, make sure the focus point stays in view.
  9. For aberrations, let's be even more fancy for the 3D. Can we slide the center of rotation and view as we zoom in? Start at 50% (middle between lens and sensor) and keep moving the center of view and center of rotation towards the sensor plane as we zoom so that the sensor remains to the right of the image and we can see the rays).
  10. Figure 9.1.6. maybe put a subtle square lattice on the sensor to stand in for pixels so that we see that things are translating when they do. But low contrast so that we see the rays well.
  11. Figure 9.1.6. are the rays uniform on the aperture? I assume you sample in polar coordinates. Make sure you properly reduce the numbers for each ring as a function of radius. Also do you properly account for the cosines on the sensor side?

Attribution note: no logged prompt names an "aberration explorer." Prompts 1–2 concern the earlier interactive aberration ray-diagram (old Figure 7.1.6); prompt 3 created a wavelength-controllable ray-trace of chromatic-aberration corrections (logged separately as Figure 22.2.3, possibly a sibling); prompts 4–11 are the current session's build of the explorer proper. Counted together here as the demo's full lineage.