28.0 PHOTO COLLECTIONS🔗⧉

Until now an algorithm has worked on one image, or a handful captured on purpose. This part changes the unit of analysis to the collection: your camera roll, a shared album, a city's worth of tourist photos, or the visual web. Two things become possible at that scale that are impossible with a single picture. First, the collection is a resource: when you have millions of examples, data can stand in for modeling: a hole can be filled with a patch retrieved from a similar scene, with the database and matching rule supplying the prior (Chapter 21.3, Inpainting Using Millions of Photographs); a landmark is reconstructed in 3-D from thousands of uncoordinated snapshots (Chapter 28.5, Photo tourism); a person's years of frames are distilled into a life summary (Chapter 28.6, Photobios, Chapter 28.4, Life logging cameras); a query is even geolocated by what the rest of the web looks like (Chapter 28.8, Pix 2 GPS). Second, the collection is an object to be organized and seen: stitched into Chapter 28.1, Photo Mosaics, searched by content (Chapter 28.2, Retrieval), curated down to the few frames worth keeping (Chapter 28.3, Auto curation), averaged into prototypes (Chapter 28.7, Average Explorer), tuned into Chapter 28.9, Personalized priors, or arranged for Chapter 28.12, Displaying images together.

The recurring point: at this scale, more data often outperforms more hand-built modeling, but collections are never neutral, so what a collection contains, and what it leaves out, quietly authors the result. The part runs from the practical (mosaics, retrieval, curation) through the data-as-prior reconstructions (inpainting, photo tourism, photobios) to the more reflective chapters (Chapter 28.10, Artistic projects with photo collections, Chapter 28.11, Pareidolia), which ask what a collection of images can do that a single image cannot.

Contents of this part

• 28.1 Photo mosaics
  • 28.1.1 The tiling-and-matching pipeline
  • 28.1.2 Color correction and avoiding repeats
  • 28.1.3 Multi-scale and irregular tilings
  • 28.1.4 Why it resolves into the target at a distance
• 28.2 Retrieval
  • 28.2.1 Classic CBIR: histograms and the text-retrieval analogy
  • 28.2.2 Deep retrieval: learned embeddings and CLIP
  • 28.2.3 Retrieval at scale: approximate nearest neighbor
  • 28.2.4 Mining what makes a place distinctive
• 28.3 Auto curation
  • 28.3.1 Technical quality: the easy rejects
  • 28.3.2 Aesthetics: the hard, learned part
  • 28.3.3 Grouping, summary, and diversity
  • 28.3.4 Arranging the collection after culling
• 28.4 Life logging cameras
  • 28.4.1 The devices and how they fire
  • 28.4.2 The memory-prosthesis reframing
  • 28.4.3 The big-data problem passive capture creates
  • 28.4.4 Privacy and ethics
• 28.5 Photo tourism
  • 28.5.1 Structure-from-motion on internet collections
  • 28.5.2 From reconstruction to experience
  • 28.5.3 Lineage
• 28.6 Photobios
  • 28.6.1 Align and order the collection
  • 28.6.2 Let the data fill the gaps
  • 28.6.3 Collection as experience
• 28.7 Average Explorer
  • 28.7.1 Galton's composite portraiture: what it was for
  • 28.7.2 Alignment is everything
  • 28.7.3 The average of a category
  • 28.7.4 An artistic lineage
  • 28.7.5 AverageExplorer: averaging made interactive
  • 28.7.6 What an average is, and what it is good for
• 28.8 Pix 2 GPS
  • 28.8.1 Geolocation as retrieval over a geotagged corpus
  • 28.8.2 The answer is a distribution
  • 28.8.3 Mapping the collection itself
  • 28.8.4 The learned successors
• 28.9 Personalized priors
  • 28.9.1 Personalized restoration
  • 28.9.2 Personalizing generative models
  • 28.9.3 The bargain and its ethics
• 28.10 Artistic projects with photo collections
  • 28.10.1 Statistical collage: Salavon
  • 28.10.2 Anticliché camera
• 28.11 Pareidolia
• 28.12 Displaying images together
  • 28.12.1 Selection
  • 28.12.2 Layout
  • 28.12.3 Color and coherence
  • 28.12.4 Themes

▸ full collapsible outline of this part