45.30 The Computational Photographer🔗⧉

Alongside this book, Frédo Durand keeps a blog, The Computational Photographer, on cameras and imaging, on research and the people who do it, and on where photography is heading. The posts are archived here as sections, reproduced with their pictures and, where a post shared slides, the deck embedded and hosted with the book. The live blog is at thecomputationalphotographer.net.

These are dated archival posts: predictions, product descriptions and personal opinions refer to the time of writing. They are preserved here as part of the book’s history.

The posts, newest first:

• Faculty jobs: why and how to get them
• Can academia compete with the resources of industry?
• Family in a box
• Do good research, run a great lab
• AI image synthesis is the biggest innovation in image making since photography
• Lidar for autofocus
• TedFest 2022
• I want a light field camera
• Lloyd Chambers's feature Christmas list
• Ethics and computational photography
• The DSLR will probably die. Are mirrorless the future of large standalone cameras?
• Eurographics keynote slides
• Halide talk
• Do good research
• Slides from my Siggraph talk about computational photography
• No telephoto on cell phones
• A perfect flat lens with metamaterials
• Why video compression fails on confettis
• This camera shows you a similar picture instead of the one you are taking
• Project idea: automatic composite portrait
• Lytro Cinema Camera: guestimates of characteristics
• Auto white balance for flash and LEDs
• Autofocus noise and microphone array

45.30.1 Faculty jobs: why and how to get them🔗⧉

June 2023.

Slides for an information webinar about faculty jobs, from Fall 2022.

45.30.2 Can academia compete with the resources of industry?🔗⧉

June 2023.

Slides from a CSAIL 60th-anniversary talk, a historical perspective from computer graphics on whether academic research can still compete with the resources of industry.

The deck is large, so it is kept on Google Drive rather than bundled with the book: slides on Google Drive.

45.30.3 Family in a box🔗⧉

January 2023.

I decided to give a try to the classic "family in a box" genre for our holiday card. It turned out better than I expected, and I wanted to share the one factor where I feel I lucked out compared to some other examples I have seen: the size of the box. I see many cases where people use a box big enough to have at least one picture with the full family. That too often means other cells end up too empty. Our box was small enough that the final result looks dense enough. Yes, fitting the two parents was pretty tight as you can see, but that only adds to the charm in my opinion.

The rest is a combination of lots of input photographs and some amount of time in Photoshop (hours and hours, probably more than 20). Lots of layers and layer masks. The final file is 12GB and a little unwieldy: saving takes forever and needs to be done often, since Photoshop crashed a few times under the load. For masking, many pictures had elements beyond the box (feet at the bottom in particular), so I usually tried "subject selection" plus the rectangle of the box. In maybe a third of the cases it did not work perfectly and I did manual paintbrush work on the layer mask.

The pair where I hold Mailys upside down was done with two photos, and I took a ton of versions of the upper one with slightly different leg positions, hoping one would match the lower one. We tried the same strategy for Laelia pouring water on herself (upper left), but it was less successful and I had to do some puppet warp and masking that took me two hours. The pair where Neva gives the red stuffy to Teo is a bit awkward in the way they hold it, but that is not Photoshop's fault, it is just a weird setup.

The perspective is of course wrong, since each box is shot straight on and has a different virtual viewpoint in the final image. The individual images are cropped from 36mm ones, probably about a 50mm perspective. Lighting is not consistent either, and is fixed with respect to each box (two soft boxes on each side plus a bit of flash from the camera). The background, mostly visible around the edges, was created with content-aware fill (patch match).

45.30.4 Do good research, run a great lab🔗⧉

September 2022.

Slides from a seminar I gave to a group of postdocs about doing research and running a group.

45.30.5 AI image synthesis is the biggest innovation in image making since photography🔗⧉

June 2022.

I will confess that I used to be skeptical and uninterested in research on image captioning and image synthesis from captions. But tools such as DALL-E 2, Imagen, and CogView, and their creative use by people such as Aaron Hertzmann, have convinced me that this is probably the most transformational change in image making and the visual arts since the invention of photography. And yes, I mean photography 200 years ago, not just digital photography.

Yes, these tools are still limited, but so was photography at first: multiple minutes of exposure for a grainy and blurry picture. Yes, these tools can be perceived as a mockery of what it means for a human artist to create a picture, and they completely bypass much of what we consider to be the creative image-making process. But the same can be and was said of photography.

Fundamentally, both photography and AI image synthesis completely change how humans envision and create images. With photography, all it takes is to point a device at a real scene and you have an image. You can complain that this is a crude and uncreative way of making an image. You can complain that you lose a ton of flexibility and artistic control. But artists quickly figured out all the creativity that can be put into making photos, and how to control the output. Making images is not only about artistic endeavors; it also enriches the human experience whether or not it can be called art, from illustrating important events to anchoring dear memories. Photography never completely replaced "manual" art, but it let us gather images that would never be possible with older manual techniques, from casual pictures of our families to photojournalism.

In the case of AI art with tools such as DALL-E 2, Imagen, or CogView, all you need to do to create a picture is write a text prompt. Similar to the switch from paintbrushes and human artistry to pressing a button on a device, this appears to be a crude interface for visual creativity, with an unacceptably low control over the final result. But like with photography, people are already figuring out how to craft prompts to get creative results. And like with photography, it will allow scenarios that were impossible before. For example, rather than use stock footage, you can write a prompt to illustrate an article, as I did for this post. That is certainly not less creative than searching an image bank, and it probably gives you more artistic control. You can write a story and get it automatically illustrated. You can author new interactive narratives where user decisions affect the storyline and get illustrated automatically. We will be able to synthesize images of memories we failed to photograph at the time. That is just the surface.

These tools raise ethical issues for sure, including their use for offensive, nefarious, or non-consensual imagery, as well as the biases and stereotypes they may inherit from their training data. But that does not make them less transformational.

This post does not discuss whether these methods are also big advances in AI, or what level of intelligence they may exhibit. I am only looking at it from the perspective of a human who wants to make images, and from that perspective this is a transformational step.

45.30.6 Lidar for autofocus🔗⧉

June 2022.

I have long wondered when and how 3D sensing technology would be involved in photography. It is interesting to see high-end videography use lidar for autofocus. See the new DJI RS3 Pro gimbal.

Update: Miki Rubinstein reminds me that the iPhone 12 switched to a lidar and uses it for autofocus as well, and that older iPhones used IR sensors and illumination for depth sensing (for Face ID, for example).

45.30.7 TedFest 2022🔗⧉

June 2022.

I was privileged to speak at TedFest to celebrate Edward Adelson's birthday and his incredible contributions to science in general, but also to the lives of many of us. Here are my slides explaining how much I owe to Ted and his ideas for my own career.

45.30.8 I want a light field camera🔗⧉

February 2020.

I want a high-end interchangeable-lens light field camera, and I want it now. But the resolution, you tell me. But the bandwidth. First, it is clear that if you look at the pixel size for phone cameras and make a full-frame sensor out of it, you can get all the resolution you need (a 24x36mm sensor at 1 micron gives you 864 megapixels). But let us say I am reasonable and only ask for, say, a 120-megapixel full-frame sensor. That is only twice the number of pixels of the Sony A7r4, and Canon made a prototype of a 120-megapixel sensor ten years ago. Even with current bandwidth (the Sony shoots at 10fps), I would be able to shoot at 5fps, not so bad unless you are doing serious sport photography.

Let us say I spend my 120 megapixels on 3x3 microlenses. My spatial resolution becomes 13 megapixels, which is really not bad (and you can probably reconstruct higher-resolution images with light field super-resolution tricks, say 20 megapixels). With these 3x3 aperture sub-pixels, I can, first, correct for optical aberration (chromatic and spherical in particular, but also field curvature) and, second, refocus by a factor of roughly 3x, meaning that if I shoot at f/2.8 I can refocus within the depth of field of an f/8 lens. This is huge.

One challenge with 3x3 is that which photosites get light depends on the aperture of the lens. It is fine if you only have f/2.8 lenses and always shoot at f/2.8; otherwise the peripheral aperture sub-pixels get less light and you are wasting them. One option around this is to only do 2x2, which is only a factor of 2 compared to Canon dual-pixel autofocus. This may end up being needed for autofocus anyway, because Canon's on-sensor AF is currently only sensitive to horizontal contrast (unlike the cross-type AF points of good DSLRs). With these quad pixels, we could refocus by a factor of 2, which is already not so bad (refocus with f/5.6 when shooting at f/2.8). It is not the dramatic refocusing you see in academic light field papers, but it can fix a lot of situations, for example when the focus lands on the nose rather than the eyes.

45.30.9 Lloyd Chambers's feature Christmas list🔗⧉

December 2019.

Lloyd Chambers, who always has good technical and user insights about photography, has a good list of features camera makers need to add to their cameras (his post). I would add the following, focusing on things that are easy and not that advanced computationally:

• A minimum shutter speed for auto-ISO as a first-class citizen, with richer control. For example, use a 1/200 minimum shutter speed until you hit ISO 3200, then accept 1/125 until a max ISO of 12800.
• Better control over ambient-light and flash balance. For example, the ability to switch between Canon and Nikon behavior (match ambient vs. keep minimum shutter speed), and in general different settings for flash vs. no flash (a different max ISO, say).
• The ability to automatically alternate settings between shots in a burst: flash / no-flash; or a slow shutter for motion blur and subject tracking followed by a fast one for a safe shot; or slow shutter with low ISO alternating with fast shutter and high ISO; aperture bracketing (at constant exposure) for depth-of-field control.
• Record gyro data for camera-shake removal, blur assessment, and video stabilization. Record accelerometer data for automatic perspective correction.
• Separate movie and still settings. I photograph and film kids: I want 1/200 for stills but 1/30 for video.
• Low ISO through frame averaging for long exposures.
• The ability to start rating and sorting photos in a way that carries over to my software (Lightroom, say).
• A simple way to cycle through faces or potential AF subjects when the camera focuses on the wrong person.
• Better ways to upload pictures to my computer. When I am home on the same wifi, they should figure out how to communicate.
• Open the software ecosystem.

45.30.10 Ethics and computational photography🔗⧉

May 2019.

Slides from my lecture about ethics and computational photography for my MIT class 6.815/6.865. They give a zoo of manipulation, both before and after digital technology and in a variety of contexts. In particular, I discuss manipulation issues in photojournalism and in the image of beauty. The slides are not meant as a self-contained discussion of ethical issues, but as illustrations to trigger questions and discussion about the topic. Students were encouraged to reflect on these issues both as citizens and as potential technology developers.

45.30.11 The DSLR will probably die. Are mirrorless the future of large standalone cameras?🔗⧉

October 2018.

People often ask me, given the improvement and ubiquity of cell phones, whether DSLRs survive. This actually entails two slightly different questions: will standalone large-ish cameras survive, and will the particular reflex design (the R in DSLR) survive? I am cautiously optimistic about the former, and very pessimistic about the latter. In this piece, I discuss DSLR vs. mirrorless. In a future post, I discuss cell phone vs. big discrete camera.

Let us see why I think the reflex design is doomed, even though it has dominated serious photography for decades. DSLR means digital single lens reflex. The term reflex comes from reflection and means that the photographer sees an optical image through the viewfinder thanks to a mirror placed at 45 degrees in front of the sensor. The mirror needs to be flipped up when taking a photograph, which, together with the shutter, is the source of the typical SLR noise. In contrast, in rangefinder or twin-lens reflex cameras, the photographer sees an image from an offset viewpoint, which can result in parallax error where the captured image is not exactly what was expected.

Historically, the reflex design has proven superior for two main reasons. First, the photographer sees exactly the image that will be taken "through the lens." This in turn makes it possible to have a rich set of interchangeable lenses, whereas it is harder to change a lens on a rangefinder because you also need to change the viewfinder lens or have marks for the field of view. Second, and this came much later, the SLR design enables superior autofocus thanks to a secondary optical path through parts of the mirror to specialized AF sensors. These sensors essentially perform stereo vision between viewpoints at the edges of a lens for a discrete set of AF points. This is often called phase-based autofocus, although stereo would be clearer in my opinion.

Video and digital cameras fundamentally changed the landscape by enabling another type of through-the-lens viewing, where the sensor that captures the final image also drives the preview, on an electronic screen rather than optically. Originally this came at the cost of inferior autofocus, because there is no space to route light towards phase-based sensors, so non-reflex cameras first used slow contrast autofocus: sweep through possible distance settings to find the sharpest one. When a DSLR is used for video, though, it must keep the mirror up, so it is back to the same constraint.

A breakthrough occurred when manufacturers made cameras perform phase-based autofocus directly on the main sensor. The idea is to split some or all pixels into two sub-pixels that each capture light from only half of the lens, so the camera can do stereo between images from the center of each half of the aperture. This explains the dramatic autofocus improvement of mirrorless systems, and it is also integrated in some DSLRs (Canon's dual pixels) because it is needed for video. For computational photography nerds, this is very similar to a light field camera, with just two sub-pixels. Dedicated DSLR AF sensors still have an edge, especially in low light, but the advantage keeps shrinking.

Sony was the first to challenge DSLR dominance at the high end with its widely applauded A7 series, and Canon and Nikon have since released full-frame mirrorless cameras. My prediction is that the DSLR is doomed, although it will probably take some time. After all, Nikon still sold a film SLR, the F6.

The mirror has to go because it is too high a cost for its diminishing benefits. It adds complexity and manufacturing difficulty, makes the body bigger, needs precise calibration, and adds a moving part that can fail. It severely constrains lens design by preventing elements from sitting near the sensor, which especially hurts wide-angle and large-aperture lenses: Canon offers a 28-70mm f/2, while the widest aperture on a DSLR 24-70 is f/2.8. And the mirrorless design makes it easy to analyze the sensor image (for face or eye detection driving autofocus and exposure) before the shot, which a mirror occludes. For video, the DSLR loses all its advantages because the mirror stays up anyway.

I do not know how long it will take. Sony became a serious player with the A7r II in 2015 and still does not have a full lens ecosystem; Nikon and Canon only have a few lenses each. But I would be surprised if, ten years from now, DSLRs accounted for a serious share of camera sales.

45.30.12 Eurographics keynote slides🔗⧉

May 2017.

Slides from my Eurographics keynote on computational photography, an extended version of my Siggraph award talk, with more concepts from the field and some suggestions for PhD research topics.

45.30.13 Halide talk🔗⧉

March 2017.

I recently gave a talk about our programming language and compiler for image processing, Halide.

Most of the credit goes to my former student Jonathan Ragan-Kelley and former postdoc Andrew Adams. As Marc Levoy summarized, with a bit of portable Halide code you can get a faster matrix multiply than Eigen, a faster Gaussian blur than Intel Performance Primitives, a faster Fourier transform than FFTW, and a faster local Laplacian than Adobe's. On top of that it runs on device and in the cloud, is supported on Linux, Windows, macOS, iOS, and Android, and compiles to x86, ARM, MIPS, Native Client, OpenCL, OpenGL, CUDA, JavaScript, and RenderScript.

Halide started as a research project, but we made the compiler open source and it is now in heavy use at Google and various other companies, where it dramatically improves programmers' productivity. For more, see the Halide paper, the PLDI paper, Jonathan's thesis, or halide-lang.org.

45.30.14 Do good research🔗⧉

November 2016.

Some thoughts about how to do research and find good problems to work on.

45.30.15 Slides from my Siggraph talk about computational photography🔗⧉

July 2016.

My thoughts about the successes of computational photography and the areas where it still has room to grow, from my Siggraph award talk.

45.30.16 No telephoto on cell phones🔗⧉

July 2016.

A conversation with Marc Levoy unlocked something that had puzzled me for a while: why do we not see cell phones with a dual imaging system to offer different focal lengths, even just a binary choice? Marc pointed out that form factor is the biggest problem. Cell phones already struggle to fit a moderate wide-angle lens in their limited depth; the iPhone 6 was criticized for its bulging lens. A telephoto would be naturally much deeper and would not fit directly. Folded optics has been attempted (the Minolta Dimage, an Apple patent, a UCSD version, and of course the Light L16) but generates its own problems, mechanical among others. At least with traditional imagers and thin cell phones, we are stuck with a wide-ish angle. Unless you follow xkcd and use a side-facing camera.

45.30.17 A perfect flat lens with metamaterials🔗⧉

June 2016.

A new Science article demonstrates an ultra-flat lens made of metamaterials that is as sharp as physically possible, has a large aperture, and high light efficiency. The work is by Mohammadreza Khorasaninejad, Wei Ting Chen, Robert C. Devlin, Jaewon Oh, Alexander Y. Zhu, and Federico Capasso from Harvard and Waterloo. They demonstrate three lenses, for wavelengths 405, 532, and 660 nm, that are diffraction-limited, have a high numerical aperture (NA=0.8), and efficiencies of 86, 73, and 66% respectively.

My understanding is that the biggest novelty is in the fabrication itself and the use of titanium dioxide, which enables high efficiency for metamaterials at visible wavelengths (previous metamaterial approaches suffered from significant light loss in the optical range). The optical property of a material derives from the way electromagnetic waves interact with its atoms, which limits the behavior of natural optical materials. The idea of metamaterials is to build composite materials from structures much smaller than the wavelength, to finely engineer how waves propagate. Early attempts worked at millimeter wavelengths (microwaves), using microstructures such as split-ring resonators. The present paper uses a simpler structure, waveguides formed by "nanofins" of titanium dioxide (tall compared to their width) deposited by electron-beam lithography.

This flat lens should not be confused with a Fresnel lens (as in a lighthouse), which flattens a refractive lens by cutting it into rings, nor with a Fresnel zone plate, which uses diffraction through alternating opaque and transparent rings. The metalens presents no spherical aberration and is essentially perfect at a given wavelength. It does suffer from chromatic aberration, which is probably the leading cause of optics complexity, so it is a little unfair to compare its size to a Nikon microscope that does fight chromatic aberration. That does not make the result any less remarkable: for monochromatic imaging, these lenses are incredibly flat and about as perfect as it gets. The authors note that chromatic aberration could be addressed with dispersive phase compensation, which would be the full game changer for photography if both spherical and chromatic aberration could be controlled in a single flat element.

Metamaterials are exciting because they vastly expand the degrees of freedom when designing optics. Traditional spherical lenses can only vary the index of refraction and the curvature of each surface, so the propagation inside is simple and isotropic, which makes spherical aberration unavoidable. Aspheric elements control the surface orientation at each point; GRIN optics varies the index within the element; diffractive optics controls light through flat binary surfaces. Metamaterials finely control the electromagnetic properties of the material below the wavelength scale, giving tremendous control over a wave. The world of optics is changing with new materials, computation, and fabrication: glass grinding is losing its dominance, most cell-phone optics are injection-molded plastic, the new metalens can be made with semiconductor technology, and 3D printers can now print optical materials.

45.30.18 Why video compression fails on confettis🔗⧉

May 2016.

A short pointer to an excellent educational video on why the independent motion of many small elements (confetti) is so hard for video compression, which relies on predicting each frame from the previous one.

45.30.19 This camera shows you a similar picture instead of the one you are taking🔗⧉

May 2016.

I love cameras that do not give you a picture of what you are looking at but instead produce a derivative. This camera performs an image search and shows you the closest match to what you are shooting. It is a cousin of the camera that refuses to take an unoriginal picture, and of the camera that prints a text description instead of an image.

45.30.20 Project idea: automatic composite portrait🔗⧉

May 2016.

There has got to be a fun computational way to do this automatically. Via Petapixel: famous faces recreated from bits and pieces of stock portraits.

45.30.21 Lytro Cinema Camera: guestimates of characteristics🔗⧉

May 2016.

The Lytro Cinema camera looks closer to reality than I expected, and No Film School got a prototype demo with editing software. Here are some back-of-the-envelope guesses about its characteristics.

The sensor is huge, more than half a meter across, about ten times larger than full frame, holding 755 megapixels at roughly 10-micron pixels (comparable to a moderate-resolution SLR), which suggests it could be shrunk. The large pixels probably give the ridiculous low-light performance and the claimed 16 stops of dynamic range; a sensor that size is beyond a standard silicon wafer, so it is likely tiled. Assuming the 100mm aperture corresponds to a 500mm focal length (a 35mm-equivalent wide angle) gives f/5, or f/10 at a 70mm equivalent, which partly offsets the shallow depth of field of such a huge sensor. Under those assumptions the depth of field would match an f/0.5 lens on a full-frame SLR without any computation.

For the microlens resolution, a reasonable guess for 755 megapixels at native 4K is roughly 10x10, though a manufacturer might prefer higher, and the packing is hexagonal rather than a grid. Using only the central microlens (about twenty times smaller than the full aperture) gives roughly f/10-equivalent depth of field, medium depth of field for cinematography, and refocusing lets you correct moderate misfocus within that f/10 range rather than dramatic rack focus. Bandwidth is the real challenge: 755 megapixels at 16 bits and 300fps is close to half a terabyte per second, and even compressed frames are hundreds of megabytes.

45.30.22 Auto white balance for flash and LEDs🔗⧉

May 2016.

White-balancing a flash or a lighting setup to the color of the ambient light makes a big difference, and I have found it surprisingly frustrating to do. You have to guess the color temperature of the artificial light, which has become tricky with the variety of bulbs. This is one area where cell phones are ahead of SLRs, since many have two LEDs matched to daylight and tungsten and balance between them automatically. Finally a set of LED panels has an auto white balance feature: press a button and it matches the ambient light. See this fstoppers post or the Kickstarter.

45.30.23 Autofocus noise and microphone array🔗⧉

May 2016.

I have been frustrated by the autofocus noise the Sony 24-70 f/2.8 records into video. Rather than resort to an external microphone and manual focus, I would love a computational audio solution: an extra microphone mounted on the lens to isolate the autofocus noise from the desired audio, and subtract it. One could even imagine doing it solely from the autofocus control information, with a little training and calibration.