An interview with Pranav Mistry, the genius behind Sixth Sense

Posted by: Emily McManus


Pranav Mistry is the MIT grad student behind Sixth Sense, a tool that connects the physical world with the world of data. He and his advisor at the MIT Media Lab, Pattie Maes, unveiled Sixth Sense at TED2009, and the Sixth Sense demo premiered yesterday on — and in both places, it has fired people’s imaginations. The TED Blog spoke with Pranav this morning, to ask him some questions that have arisen on and at the TED office. From the interview:

Why choose a projector versus goggles?
We actually thought a lot about this. At MIT, lots of research has been done with glasses — there’s even research going on to put information in your contact lenses. But this particular project has an important aspect: We want this thing to merge with the physical world in a real physical sense. You are touching that object and projecting info onto that object. The information will look like it is part of the object.

Read the full interview with Pranav Mistry, after the jump >>

Watch the Sixth Sense demo on >>

TED Blog’s interview with Pranav Mistry, 3/11/2009

What was your role on the project?
I took this from the idea, the concept, and developed the software and the hardware. Pattie is my advisor. She helps me brainstorm, “What should we do next?” From the beginning, I started working on the concept of merging the physical world and the digital world, like with my earlier project, Quickies, merging physical sticky notes with digital data.

Why choose a projector versus goggles?
We actually thought a lot about this. At MIT, lots of research has been done with glasses — there’s even research going on to put information in your contact lenses. But this particular project has an important aspect: We want this thing to merge with the physical world in a real physical sense. You are touching that object and projecting info onto that object. The information will look like it is part of the object.

Also, a projector opens up interaction and sharing. Say you and I are walking down the street in New York, talking, and suddenly I get mail we both want to see. I can show it on a wall, and we can take a decision together right there. It’s opening information to share. At a coffeeshop, we can use the whole table as an interactive surface.

Where’s the battery?
The project itself contains a battery inside, with 3 hours of battery life. The other thing is, I’m making a small solar panel. I’m trying that out right now because I want to go with sustainable energy — and so you don’t always need to be charging. Whenever you’re outside, you’ll be charging, and with the system, you can be outside more.

How does the software know what you want the system to do next?
The software works on the basis of computer vision. There’s a small camera acting as your eye, your third eye, your digital eye, connecting you to the world of digital information. Processing is happening in your mobile phone, and basically works on computer vision algorithms that we developed ourselves, taking advantage of some open-source code but mainly writing code ourselves here at the lab. We had to write a lot of algorithms from scratch here because there was nothing that did what we wanted. We wrote 50,000 lines of code.

The software recognizes 3 kinds of gestures:
+ multitouch gestures, like the ones you see in Microsoft Surface or the iPhone — where you touch the screen and make the map move by pinching and dragging.
+ what I call freehand gestures, like when you take a picture [as in the photo above]. Or, you might have noticed in the demo, because of my culture, I do a namaste gesture to start the projection on the wall.
+ iconic gestures, drawing an icon in the air. Like, Whenever I draw a star, show me the weather. When I draw a magnifying glass, show me the map.

You might want to use other gestures that you use in everyday life. This system is very customizable. Because it’s my choice, I do it my way in the demo, but I don’t want the user to change their habits. I want the Sixth Sense to change for them.

Have you thought about using this device for gaming?
Definitely. We can do all the kinds of gaming that exists now, but not only that, we can use the physical world inside the game. You can play with physical stuff, invent some new games. Maybe you can hide something in the physical world — open a book and hide something in the pages.

We’ve been using “Minority Report” as shorthand to explain the device, or the heads-up screen in “Robocop.” But was this device influenced by science fiction
I’m not a very big fan of science fiction. I think that I’m a very big fan of living in the physical world. I’m good with digital technology, but I start to miss the physical world. I miss riding my bike, talking to friends. Technology now separates us from the physical world more and more. Even social networking sites are taking us away from the physical world.

At the lab, we like making things that we can touch, we can feel, we can take with us wherever we want to go, that we know how to interact with. The digital world has power because it has dynamic information, but it’s important that we stay human instead of being another machine sitting in front of a machine.

Whatever science fiction movies we watch now, we can make the technology real in two days. What we can do is not important. What we should do is more important.

What would you have said on the TED stage, if you and Pattie had had one more minute?
We have lots of applications. Any application now on your computer or mobile phone, you can use in Sixth Sense. In fact, I was going to come on the stage and do a live demo, but then we decided to use the demos I had filmed. We were worried about technical problems and the low light. But we have more to show.

The demo has gotten a lot of attention. What’s life been like since your demo at TED?
As soon as the talk finished, so many people rushed up to us. Before TED I was working, easy, three months straight in my lab. I was thinking after TED I would take a one-week break. But even today I haven’t taken that one-week break. We are getting, every day, 50 mails, “we should do this, we should do that.”

There’s some really interesting comments, not only from people interested in making computer interfaces, but people asking, “Why can’t we use this system for people who have accessibility problems, blind people, deaf people?” The camera can act as a third eye for the blind person, and tell them what it sees. It could be an ear for a deaf person. Ideas are also coming from developing countries — in part because of the low cost. It cost me $350 to build Sixth Sense in the lab, but the price will come down.

Comments (51)

1 2 3
  • Andres Ramos commented on Mar 29 2010

    really is so great!!… he is a real genious… in conclusion “the mind hasn’t limitation”, “nothing is imposible”. Congratulations Pranav!

  • melvin diaz commented on Mar 26 2010

    When i first saw the video in one video sharing site, I didn’t believe it at first because I think the technology today is not yet capable to do that. Maybe in the next 10 to 15 years, but not now yet. But Pranav made it happened. Sixth sense is the biggest breakthrough in this century on my own humble opinion! Connecting the Physical world with the digital world is astonishing!

  • hailey knowlton commented on Mar 25 2010

    Very Impressive device. It is totally amazing that this kind of technology exists!!

  • krishna sai commented on Mar 13 2010

    guys can i buy this and how much does it cost?

  • Mohammad Hosseini commented on Mar 3 2010

    Well, nice try! congratulations to Pranav, the MIT grad student and his work team.
    Technologically impressive sure, but that’s no reason to think it’ll enrich our lives.It really needs more support and development…the future is now !

  • Mohammed AL-owaini commented on Feb 2 2010

    I think this is good and an amazing invative and they have many advantages and disadvantages for this new tecnology.For example, the most intersting advantages which is you can know the person who is he, for example, teacher, student or doctor.

    The other disadvantage which is not good for health because this invative it can effect your body. For example, when you put mobile in your pokect it can effect your body.

  • KESAVAN VR commented on Jan 12 2010

    first of all congratulations yaar…. this will certainly change the entire mindset of people regarding machines and technology……moreover it will also help physically handicapped to live a better life….now i know why they say technology changes life……. congrats once again

    what are the requirements ie on the infrastructure part for implementing this technology ?

    Is India having the infrastructure required ?

    can this technology be used as complete alternative to computers ?

  • mamatha raghu commented on Jan 8 2010

    sixth sense technology …simply mind blowing thing…this guy is very amazing and wish him all success for all his projects ..

  • Srinivasa Parupalli commented on Dec 3 2009

    I was just amazed and watched the video for may be 20 times continuously and forwarded the link to more than 200 hundred people..that I know.. This is really a great break though in digital technology.. I liked your answer to One of the questions , “What we can do is not important. What we should do is more important.” wish you a great success in all your future projects. I wanted to ask the same question as Hasit Mistry, and also what kind of mobile phone and speed should we have to achieve the results? – Cheers

  • VINOD K V commented on Nov 17 2009

    Amaaaaazing is not enough to describe this…may be a threshold where reality and fantasy merge….to may be conquer the mysteries of life and the Universe paving the way for man as Creator Ultimate?

  • Bharat Mistry commented on Nov 10 2009


    It is simply fantastic you have proved your-self

    I want something more to be added in your project.

  • John David (JD) Walsh commented on Nov 8 2009

    this guy is amazing!

  • Ritwij Sase commented on Oct 29 2009

    That is the real future. Hope more ideas’ll come out of your mind.

  • gaurav bhardwaj commented on Jun 23 2009

    i think it’s a greatest invention in the field of science,and i think it could be used to blind people,as they can’t see, but they know shapes,it can act as their eye in this world.
    all the best, pranav.

  • Mark Robinsson commented on May 7 2009

    Sixth Sense is an extremely interesting project.
    Wish luck to Pranav.

  • Handy Yuspandi commented on Apr 17 2009

    I hope this technology will enable us to experience the world in better way, with little loss possible to what we perceive important like gut feeling, first impression, the thrill of being surprised when we buy unknown product, and so many irrational qualities some of us think important part of being human. Like J.J. Abrams and his mystery box.

    Thank you to show us the amazing possibility of our world tomorrow. Keep it up!

  • alan buchanan commented on Apr 2 2009

    Sounds like some cool technology, i’ve never heard if it before.
    Man I must be getting so out of touch.

    I’ll be keeping a close eye on it now i think.
    Alan – my internet business

  • William Loughborough commented on Mar 15 2009

    Because this is so vision-centered, I would hope that you at least consider how it will be accessible for people with low/no vision.

    We are so beset with “retinal chauvinism” that we often fail to look at the significance of these matters in anything we do.


  • Hasit Mistry commented on Mar 15 2009

    Congratulations on the success of the project and I really hope it comes out in the market as soon as possible.
    What I would like to ask is that, where does the information come from? Is it stored in the device (which I don’t think is possible) or does it come from the internet? And if it’s the internet, there must be a database. Does it search the whole internet for information on only one kind of toilet paper?

  • Nilesh Vyas commented on Mar 26 2009

    Congratualations on the brillient success of sixth sense… Pranav you are the proud of Gujarat

    love ever !!
    …………………………………………………………………………………….. | |

1 2 3