Your studio explores new ways physical objects relate to digital information. Why is that mission important to you?

The way the human body and human mind are set up, we’re incredibly good at using physical objects and interacting with them. That physical world is so rich, yet it’s often neglected in the traditional ways we interact with computers — such as a keyboard and mouse, or even a touch screen. Whatever you’re doing, it feels the same, whether you’re writing a spreadsheet or a love letter. There’s no extra texture, or smell, or any other sensation, besides the visual element. It would be a shame to just totally ignore those other aspects of the physical world.

That’s why I use new technologies to bring back some of that physicality into the digital world. Sometimes the goal is purely functional, like helping somebody in an office do something faster — the sense of touch gives them this feedback that helps them do things more quickly. Sometimes the outcome is simply to be more fun — a richer experience. I try to integrate ways one can interact with computers that are faster, easier to understand, more social, and more fun. That’s something that’s been pretty difficult to do with computers in the past.

What are the kinds of projects you work on at your Patten Studio?

One of my earlier projects, the “Sensetable,” I talk about in my TED Talk. The Sensetable has a circuit built into the tabletop surface, which tracks the movement of objects on the table. My favorite application of it is one where the objects on the table represent different atoms. You can bring them together to cause a “chemical reaction.” That project is now an exhibit at the museum of Science and Industry in Chicago.

Another interesting piece was the Gravity Harp for Bjork. Some colleagues here in my co-working space asked me to help them with the harp, because in order to get it to actually play music, they needed some very complex control of the motion of the instrument. The original design had 37 pendulums in a ring. That ring was 50 feet wide, and 25 feet tall.

We had a really tight production schedule, so ultimately, it was much smaller, but the core idea of pendulums that play a harp remained constant throughout the process.

Right now, we’re working on a series of windows for Barney’s New York. We recently had a series of kinetic installations with motors and televisions, commemorating Carine Roitfeld, the former editor of French Vogue.

Currently we have a display up about a French shoe designer, Christian Louboutin. One of the windows has a man’s pair and a woman’s pair of shoes. The man’s pair sort of chases the woman’s pair around the window. We built this huge mechanism to use magnets underneath the floor to actually move the shoes. These window displays are kind of a crazy combination of technology and art. The people at Barney’s are really exited about having kinetic elements in their windows, and creatively, we see things from the same perspective, so it’s been really fun working with them.

How did you first come across this idea of combining the physical and digital?

I was an undergraduate at the University of Virginia, and I was doing research in a virtual reality lab with Professor Randy Pausch. When people would wear the virtual reality head-mounted displays, they would often experience a phenomenon called “simulator sickness.” It’s kind of like getting seasick. Your eyes are giving your brain conflicting information about what you’re experiencing, compared to what the rest of your body is perceiving. The theory I heard about why this goes on is that your body thinks, “Something really weird is going on here. I must have eaten something bad, so I should throw up.” I felt like that was such a clear signal that this type of interaction was not well suited for the human body.

To help people acclimate, one thing we found worked really well was to give the person something physical to hold in their hand during the virtual reality session. We found they had a much easier time understanding their relationship between the physical and virtual world that way.

For example, we had this one demonstration called the “Light Saber Demo.” In it, the player is Luke Skywalker in Star Wars and he or she had to fight with this light saber. We would just give the person a flashlight to hold in their hand, and they would see the image of the saber on the screen. That physical object made it a much more believable story, and people felt much more comfortable.

From that, I got really excited about this idea that physical objects give us this special kind of information that you can’t get visually. Later at a conference, I saw some work out of MIT from a professor named Hiroshi Ishii that just blew me away. He had a table with little building models on top of it, coupled with video projection on the tabletop. There were these physical models, but digital information was projected around them. As soon as I saw that, I asked one of his students, “Who’s your advisor? I need to meet this person!” It was a love at first sight thing. I went to MIT and studied with Professor Ishii, and wound up doing my PhD there.

You mentioned working with Randy Pausch, who is famous for his talk on living life to the fullest. Has his philosophy affected you at all?

Randy was a very dynamic guy, and we were a very tightly knit research group under his leadership. I think that when you’re in that context, there are going to be a lot of influences. Some you notice and some you don’t notice. One of the most important lessons I learned from Randy is that time is very valuable. He has another talk that he used to give often, his time management talk. That’s a theme that is there in both of his well-known talks: time is the most valuable resource we have. At any moment, you have to think, “Out of all the things that I could be doing, is what I’m doing right now the right way to spend my time?”

Being around him really inspired me to do the best I can to make sure the work I’m doing is pushing things in the direction I want to go.

Your work combines design, electronics, software, and requires picking up new skills on the fly. Have you discovered anything surprising about this eclectic type of work?

One surprising thing is how much the skills from different disciplines transfer. For example, before I had ever worked on electronics at all, I would see people working on it onscreen, and think, “Wow, whatever that person is doing, it’s really beautiful.” The software that you use to design electronics is very colorful, because everything is color-coded. People inadvertently create these really beautiful images. And one of the things I realized, once I got better at electronics, is that often a circuit that’s better designed, that does its job better, is actually visually more aesthetically beautiful as well.

I’m always trying to go after projects that involve different kinds of skills, and that leads to unexpected places, like doing these windows for Barney’s. But it keeps things really interesting and fun. When we started this window for Barney’s, we were given these shoes that were $1,000 dollars each, and we had to drill holes in them. I thought, “Wow, I’ve never had to drill a hole in a $1,000 shoe before.” I feel like I’m always getting to do something new. Sometimes I stop and think, “How did I get here?” I’m happy to be here, but it’s certainly not where I thought I would be 10 years ago.

How has the TED Fellowship affected your work?

Being a TED Fellow has given me the opportunity to connect with a lot of people. Some have been face-to-face interactions at the conference, while others have been through the TED website. For example, at the conference I met a variety of people who have offered to help me or collaborate, and the most recent person I hired at Patten Studio learned about my work through TED.

Also, the TED Fellows participated in a pre-conference, which was extremely useful. I learned a lot about how to communicate about my work with the press and with the public.

Finally, it’s also been very valuable to have this group of peers in the TED Fellows. The group is so supportive and people within it have so many different perspectives to offer. There is a lot of trust in the group and a genuine willingness to help each other achieve big goals.

In general, work at my studio has picked up a lot since I was selected as a TED Fellow. We’ve gotten more interesting projects and more of them.

Part of your approach at Patten Studio is to think of the user experience first, and think next of the technology. Doesn’t it often prove impossible to make the technologies match up to the vision?

Well, it’s true that there are lots of technologies that I would love to have that don’t exist. Like the ability to track objects or people in 3D space without using batteries. There are lots of ways to solve problems like that in a limited scope, but there’s no one off-the-shelf solution that solves the problem in the general case.

But a lot of times, if you come up with the user experience that you want to have, and you realize that the technology to do it doesn’t exist, maybe you can create that technology. I think that’s the most exciting outcome, because it’s using technology to expand what’s possible in the world of design.

There’s a list of technologies that people keep generally thinking, “Well, if we had this technology, we could do all these amazing things.” One that people think about a lot is digital clay. It’s the idea that you could have clay that would remember its shape, and then you could program it and get it to do something like transform itself into the shape it had yesterday. These kinds of technologies are really exciting, especially for people who design 3D structures like cars or buildings. No one really has any idea how to make digital clay. But with technologies like these, there are people who recognize that the vision is really exciting, and they start taking baby steps towards that idea. Often, they eventually get somewhere really close that’s useful for a lot of things.

What kind of not-yet-existing technologies are you taking baby steps towards right now?

Well, one idea that I’ve been excited about for a long time is an outgrowth of my Sensetable. I built another similar table that not only tracks the objects, but also moves them. To do this, I used electromagnets, which are too power-hungry for every day use. So I’m working on a more generalized, practical technology to do the same job. I’d like to create a bunch of small objects that you can put on any tabletop, where they can drive themselves around. By moving themselves to different orientations in front of you, they become the user interface: they become the way you interact with the computer.

For example, let’s say you’re editing a video. These knobs would move out onto the table in front of you, and arrange themselves in a way that makes sense for that particular task. Then, if you’re going to draw a picture, they rearrange themselves in another way. You have this constantly evolving palette of tools at your disposal. You can have this kind of give-and-take, tug-of-war interaction with the computer, where, if you’re doing something that you’re not supposed to do, you can sort of feel the computer pulling against you as you push an object on the table.

I’ve made the proof-of-concept of this, in the context of the magnetic array, but I’d like to make this something that you could put in your pocket, and pull out and use on any kind of tabletop surface.

There are many aspiring social entrepreneurs out there who are trying to take their passion and ideas to the next level. What one piece of advice would you give them, based on your own experience and successes? Learn more about how to become a great social entrepreneur from all of the TED Fellows on the Case Foundation’s Social Citizens blog.

It’s important to be honest with yourself about what you’re good at and not good at. Team up with other people whose strengths are complimentary, in order to form a well-balanced team. I think that’s one of the most important things. It’s tough to go about starting a business by yourself, and it’s much more fun, and better in a lot of ways, if you can do it as part of a team.