Live from TED TED Conferences

How a silent video can reveal sound: Abe Davis’ knockout tech demo at TED2015

Posted by:
Abe Davis speaks at TED2015 - Truth and Dare, Session 1. Photo: Bret Hartman/TED

In Session 1 of TED2015, Abe Davis demonstrated an imaging technology that pulls sound from silent video. Photo: Bret Hartman/TED

Abe Davis begins his talk in Session 1 of TED2015 by showing what appear to be two still images — a closeup of the inside of a wrist, a sleeping baby in a crib. But these are not photographs, he says; they are videos. Even if we can’t see them, there are tiny motions in the frame: the blood pulsing, the baby breathing.

“We created software that finds the subtle motions in video and amplifies them so they become big enough for us to see,” Davis says, showing the images again, this time processed to magnify the tiny motions in the video. Now it’s easy to see a pulse in the wrist, to see the baby’s chest lift and fall.

The team led his team to another question: Could they use this video technique to see sound? Sound causes all objects to vibrate, Davis reminds us. “What if we record those vibrations with a high-speed camera and use software to extract tiny motions, and see what made those sounds?”

In an early experiment, Davis and his team filmed an image of a plant at thousands of frames per second — while playing “Mary Had a Little Lamb.” The music shook the leaves imperceptibly. But by analyzing the high-resolution video and extracting the tiny movements, then extrapolating what might have caused those movements, the team was able to extract an only slightly degraded version of the song. How is this possible? As Davis says, the music moves the leaves about a micrometer, or one thousandth of a millimeter. But taken en masse, these tiny movements “add up to something perceptible.”

Abe Davis speaks at TED2015 - Truth and Dare, Session 1. Photo: Bret Hartman/TED

A specialized camera is set up to film a potato chip bag through a piece of soundproof glass — and the vibrations of the bag reveal words spoken nearby. Photo: Bret Hartman/TED

For another demo, the team set up a specialized high-speed camera behind soundproof glass, and spoke the lyrics of “Mary Had a Little Lamb” in the vicinity of an empty potato chip bag. Again, they were able to recover sound — this time, spoken words. They filmed a pair of earbuds plugged into a computer playing music; from the video, they extracted audio so clear they could Shazam it to identify the song.

The next question: Could they do this with video equipment that didn’t cost tens of thousands of dollars? The team modified their algorithm to use the rolling shutter of an ordinary camera to detect tiny, invisible motions. And again, from a recorded bag of candy: “Mary Had a Little Lamb.”

Before he continues, Davis wants to acknowledge a concern. “People hear about this technology and immediately think about surveillance,” he says. “Keep in mind, there’s already a lot of mature technology out there for surveillance.” (As he tells Chris Anderson later in a Q&A, “people have been using lasers to eavesdrop for decades.”) What’s more exciting to him: Building a new lens to see the way the world works.

From here, Davis wants to show us something a little different — something that he’s only been working on for a few months. He was curious: Could we use this video technique to learn about the physical properties of an object, to predict how it would move? Though this new tech is yet unpublished, Davis gives us a sneak peek at a video technique that analyzes how an object moves, then generates a clickable image that moves just like the actual object. (We promise to bring you this video as soon as we can.)

Chris Anderson introduced this talk by saying that we were about to see a technology that could be as game-changing as Jeff Han’s touchscreen demo in 2006, a year ahead of the iPhone. Davis stresses the potential here too. “We’re just starting to scratch the surface with what we can do with this type of imaging,” he says. “It gives us a new way to capture our surroundings.”

This talk is now available on Watch now »

Abe Davis speaks at TED2015 - Truth and Dare, Session 1. Photo: James Duncan Davidson/TED

In a theater whose design evokes ripples of sound, Abe Davis demonstrates the future of listening. Photo: James Duncan Davidson/TED