Dr. JoAnn Kuchera-Morin works on the AlloSphere, one of the largest scientific and artistic instruments in the world. Based at UC Santa Barbara, the AlloSphere maps complex data in time and space. Dr. Kuchera-Morin, a composer, demoed the AlloSphere at TED2009 in February, showing five films of scientific data mapped visually and sonically into compelling art. Last week I talked with Dr. Kuchera-Morin about the AlloSphere — what it does, how it works, who uses it, and how you turn raw data into sound. From the interview:

Some of my mathematician colleagues are working with 6-dimensional figures. What happens when your math starts to get so complex that you can’t draw it by hand anymore? Scientists have such tremendously rich math data that the instruments they use now can’t actually see it. You get measurements from it, but can you take those math coordinates that describe it and map it visually and sonically?

There are scientists who have lost the ability to perceive their data. Now they might have the ability to perceive this data again through portals that let them see and hear their data, not just see a string of numbers.

Read the full interview >>

More: If you’re around Santa Barbara next week, hear JoAnn Kuchera-Morin’s work at the Primavera Festival

More: For a celebration of boundary-breaking science research, read “In search of the black swans,” Physicsworld April 2009So what’s going on at the AlloSphere? What do you hope to use it for?

Different people from different facets understand the nature of the instrument, and understand what we’re trying to do to bring art, science and engineering together. Some come from the practical level of research, with implications for industry and development. But we’re also working with mathematical concepts quantifying things that are almost spiritual. We’re advancing the nature of who and what we are and the nature of the universe.

We’re all just out there mapping, looking for constructs.

Constructs?

We’re looking for patterns. We’re looking for beauty. The way that we appreciate beauty deals with the nature of complexity, uniqueness, subtle changes over time that catch you by surprise. It’s something we look at as artists, and our scientists are looking for this too.

I was talking to a Nobel Prize-winning physicist on campus. And he said, “Why do I need this? My work is data. It’s numbers.” And I said, “Have you ever been working on a problem on your computer screen, you’ve been really stuck, and then one of your colleagues walks through your door, and from three and a half feet away says, ‘That data doesn’t look right’?” That’s the value of looking for patterns in data. Patterns you can’t see when you’re up close.

We intuitively know these things. Cultures have been weaving these patterns, plowing these patterns, etching these patterns. This is what we are.

Tell me more about mapping data sonically.

Sonification is really important. We hear vibrations up to 20-20,000 hz, and we see up in the gigahertz range. But it’s all data, and you can take data and convert it into another form. What happens is, I take visual data, drop it 27 octaves and do the proper transformations so I can hear it. Think of a mathematical equation that can be seen and heard. If you take any math data and sonify and visualize it, you’ll be able to see and hear patterns — you’ll see balance, continuity, contrast, surprise, all the things that catch people’s attention.

Audio and visual data come with different mindsets. When I started building media systems back in the ’80s, it was from an audio point of view. In audio, you always have to be in real time — your data is mathematically moving from one point to another point. When we digitize a sound, we’re taking a snapshot of that sound and freezing it over time. Aspects of that constraint have plagued us in the sonic world.

On the other hand, animation is about taking still shots and speeding them up over time; they’re coming from a different place. I’m convinced everything has to be built from the ground up with the transitions in place, visually and sonically.

What’s it like inside the AlloSphere? Where do you sit?

You don’t sit. You’re actually standing on the bridge. The bridge is designed so you’re looking at a horizon, and your eye horizon is geared for you to be standing. We’re expecting people not to be passive viewers.

The whole instrument should be a multi-user interactive space. Researchers will be in there with gloves on, and they can reach out and grab data, pull data to them. 15 researchers can be in there interacting and doing multiple things with the data at the same time. We’re going to have to think about how to track people as they move around.

The instrument will always be used as a production instrument. Artists work this way, and so do scientists — they want to see, to observe. This could be a very large microscope for them to look into.

Something about your talk that struck me is the teamwork between artists and scientists, and the fact that artists seem to be equal partners in this work.

Between artists and scientists, the connection is so tight, a lot tighter than most people think.

All of art is mathematics. I’m a composer, and when I was writing chord progressions, there was a discipline that removed me from the math — I wasn’t aware I was using linear algebra principles. When you move up and learn more, you start learning all the mathematical constructs and moving into nontraditional mathematics. These are things that composers in the mid-’40s were doing, with these large-scale orchestral works. We’re pushing ourselves with our vocabulary, and our music is getting more and more complex. All of a sudden, that leads us to instruments like a computer to manipulate complex music. The content has been driving the technology.

Similarly, some of my mathematician colleagues are working with 6-dimensional figures. What happens when your math starts to get so complex that you can’t draw it by hand anymore? Scientists have such tremendously rich math data that the instruments they use now can’t actually see it. You get measurements from it, but can you take those math coordinates that describe it and map it visually and sonically?

There are scientists now who have lost the ability to perceive their data. Now they might have the ability to perceive this data again through computers that have portals that let them see and hear their data, not just see a string of numbers.

And this work is described as “art,” not just as a really supercharged presentation tool.

For me, why what I’m doing is art is, I’m pushing science into domains where people don’t know how to visualize and sonify that data.

In the demo, what you saw was not scientists coming in and saying, “Here’s what I think this should look like.” They came in with data and said, “This is what I think I’m going to see, but I’m not sure.” And us saying, “Well, how about if we tried it this way instead?”

And sometimes there would be resistance. We’d say, “Give us the leap of faith to try it this way, and let us see if we can see something new.” I mean, how do atoms look, how do electrons look? There are always these various ways that we can look at or hear data, and you might see something by looking in one way or another way.

I was talking to one of my scientist colleagues — I came into his lab waving a research paper and I said, “Hey, I’m trying to understand this. Help me!”

And he said, “You’re not supposed to understand this. I barely understand this.”

And I said, “You don’t understand: I’m an artist, I’m ignorant, I have no fear, and I have all the time in the world. Let me take your data and let me play.”

We are having some of the most exciting conversation on the AlloSphere bridge! It’s scary to the scientists, because when we make these big leaps of faith, we can make mistakes. You have to learn to distance yourself from your data and become a third-party observer, especially when it’s so complex and so small — how can you wrap your minds around it? The math is so complex with one electron spin, how can we understand any more than that? But we’ll build on what we have.

These are things that have the artists sparked as well as the scientists sparked. We are the Beethovens of the 21st century. We’re mapping terrain in space and time.

What’s next for the AlloSphere — when will it be ready to go? And can we come see it in person?

We’re revving up. We’re engineering the instrument — it’s about 4 to 6 months away from lights-on. We’ve got slices of it lit up right now, and have had various configurations of the whole sphere lit up. But to match the two hemispheres together is an engineering feat of magic.

Not to say it’s not lit right now. There’s research going on right now. After TED, we had half of a hemisphere lit up, 360-degree audio, 24 channels of sound.

Come see it. After TED we had a group drive up the coast to see the AlloSphere, and different groups from the TED community have come up from LA — both industry partners interested in potential use, and artists and scientists interested in using it. When they see it, they say, “Words and video can’t describe the real experience.”

More: For a celebration of boundary-breaking science research, read “In search of the black swans,” Physicsworld April 2009

More: If you’re around Santa Barbara next week, explore CREATE and hear JoAnn Kuchera-Morin’s work at the Primavera Festival

Pranav Mistry is the MIT grad student behind Sixth Sense, a tool that connects the physical world with the world of data. He and his advisor at the MIT Media Lab, Pattie Maes, unveiled Sixth Sense at TED2009, and the Sixth Sense demo premiered yesterday on TED.com — and in both places, it has fired people’s imaginations. The TED Blog spoke with Pranav this morning, to ask him some questions that have arisen on TED.com and at the TED office. From the interview:

Why choose a projector versus goggles?
We actually thought a lot about this. At MIT, lots of research has been done with glasses — there’s even research going on to put information in your contact lenses. But this particular project has an important aspect: We want this thing to merge with the physical world in a real physical sense. You are touching that object and projecting info onto that object. The information will look like it is part of the object.

Read the full interview with Pranav Mistry, after the jump >>

Watch the Sixth Sense demo on TED.com >>

TED Blog’s interview with Pranav Mistry, 3/11/2009

What was your role on the project?
I took this from the idea, the concept, and developed the software and the hardware. Pattie is my advisor. She helps me brainstorm, “What should we do next?” From the beginning, I started working on the concept of merging the physical world and the digital world, like with my earlier project, Quickies, merging physical sticky notes with digital data.

Why choose a projector versus goggles?
We actually thought a lot about this. At MIT, lots of research has been done with glasses — there’s even research going on to put information in your contact lenses. But this particular project has an important aspect: We want this thing to merge with the physical world in a real physical sense. You are touching that object and projecting info onto that object. The information will look like it is part of the object.

Also, a projector opens up interaction and sharing. Say you and I are walking down the street in New York, talking, and suddenly I get mail we both want to see. I can show it on a wall, and we can take a decision together right there. It’s opening information to share. At a coffeeshop, we can use the whole table as an interactive surface.

Where’s the battery?
The project itself contains a battery inside, with 3 hours of battery life. The other thing is, I’m making a small solar panel. I’m trying that out right now because I want to go with sustainable energy — and so you don’t always need to be charging. Whenever you’re outside, you’ll be charging, and with the system, you can be outside more.

How does the software know what you want the system to do next?
The software works on the basis of computer vision. There’s a small camera acting as your eye, your third eye, your digital eye, connecting you to the world of digital information. Processing is happening in your mobile phone, and basically works on computer vision algorithms that we developed ourselves, taking advantage of some open-source code but mainly writing code ourselves here at the lab. We had to write a lot of algorithms from scratch here because there was nothing that did what we wanted. We wrote 50,000 lines of code.

The software recognizes 3 kinds of gestures:
+ multitouch gestures, like the ones you see in Microsoft Surface or the iPhone — where you touch the screen and make the map move by pinching and dragging.
+ what I call freehand gestures, like when you take a picture [as in the photo above]. Or, you might have noticed in the demo, because of my culture, I do a namaste gesture to start the projection on the wall.
+ iconic gestures, drawing an icon in the air. Like, Whenever I draw a star, show me the weather. When I draw a magnifying glass, show me the map.

You might want to use other gestures that you use in everyday life. This system is very customizable. Because it’s my choice, I do it my way in the demo, but I don’t want the user to change their habits. I want the Sixth Sense to change for them.

Have you thought about using this device for gaming?
Definitely. We can do all the kinds of gaming that exists now, but not only that, we can use the physical world inside the game. You can play with physical stuff, invent some new games. Maybe you can hide something in the physical world — open a book and hide something in the pages.

We’ve been using “Minority Report” as shorthand to explain the device, or the heads-up screen in “Robocop.” But was this device influenced by science fiction
I’m not a very big fan of science fiction. I think that I’m a very big fan of living in the physical world. I’m good with digital technology, but I start to miss the physical world. I miss riding my bike, talking to friends. Technology now separates us from the physical world more and more. Even social networking sites are taking us away from the physical world.

At the lab, we like making things that we can touch, we can feel, we can take with us wherever we want to go, that we know how to interact with. The digital world has power because it has dynamic information, but it’s important that we stay human instead of being another machine sitting in front of a machine.

Whatever science fiction movies we watch now, we can make the technology real in two days. What we can do is not important. What we should do is more important.

What would you have said on the TED stage, if you and Pattie had had one more minute?
We have lots of applications. Any application now on your computer or mobile phone, you can use in Sixth Sense. In fact, I was going to come on the stage and do a live demo, but then we decided to use the demos I had filmed. We were worried about technical problems and the low light. But we have more to show.

The demo has gotten a lot of attention. What’s life been like since your demo at TED?
As soon as the talk finished, so many people rushed up to us. Before TED I was working, easy, three months straight in my lab. I was thinking after TED I would take a one-week break. But even today I haven’t taken that one-week break. We are getting, every day, 50 mails, “we should do this, we should do that.”

There’s some really interesting comments, not only from people interested in making computer interfaces, but people asking, “Why can’t we use this system for people who have accessibility problems, blind people, deaf people?” The camera can act as a third eye for the blind person, and tell them what it sees. It could be an ear for a deaf person. Ideas are also coming from developing countries — in part because of the low cost. It cost me $350 to build Sixth Sense in the lab, but the price will come down.

Watch the Sixth Sense demo from TED2009 on TED.com, where you can download this TEDTalk, rate it, comment on it and find other talks and performances from our archive of 390+ TEDTalks — including more demos.

Get TED delivered:
Subscribe to the TEDTalks video podcast via RSS >>
Subscribe to the iTunes video podcast
Subscribe to the iTunes audio podcast
Get updates via Twitter >>
Join our Facebook fan page >>

Subscribe to the TED Blog >>

David Merrill is a grad student in the Fluid Interfaces Group at MIT’s Media Lab. He and his fellow students in this group work on new technologies that give us more and better abilities to do things we want to do. At TED2009 he gave a demo of his main project, Siftables.

Today the TED Blog interviewed Merrill to get some details about the Siftables project — and answers to some questions that many have asked since his demo. Here’s a snippet:

I have heard so many people say: “My kids will love these. When can I get some?” The realization has been hitting us over the past few months that the potential for kid-oriented interactions is huge and meaningful.

Find the full interview with David Merrill below the fold >>

Full text of email interview with David Merrill (2/23/09):

A sound problem interrupted your live talk — but everyone noted how you kept your cool. What’s the secret?

I have been in blues and rock bands since I was a teenager, and there’s nothing like performing in front of big audiences over and over to cure any trace of stage fright. The sound problem was just like having a band-mate break a guitar string, and whenever that happened during a concert I’d have to stall for time while he fixed it and keep the audience with me in exactly the same way.

You’re in China now, looking at manufacturing possibilities for Siftables. What grabbed your interest?

The electronics market in Shenzhen is amazing — it’s a fabulous bazaar for technology geeks.

There’s something satisfying about the clatter of Siftables on a tabletop. But how hardy are they?

Siftables are pretty durable little guys. We worked with Amit Zoran (of the MIT Media Lab) and Noah Murphy-Reinhertz (of polysaturated.com) to make the plastic housings, and they have been working really well — we haven’t had any break on us yet. However, one of the things that we’ve seen at every factory is the drop-test rig, and I think we probably have a little more robustification to do before dropping heavy objects onto them.

Earlier on in the design cycle, what was something you tried that just did not work?

We initially made little charging docks — one for each Siftable. They were cute and they mostly worked, but they complicated the design of each Siftable block. We continue to learn that simple solutions are usually the best ones.

Many people are wondering whether Siftables are going to be openly programmable. What do you think?

I can’t say much about this now, but stay tuned. There are a lot of creative people out there, and we would love to enable them to design interactions for Siftables.

Let’s say you had an extra minute on stage. Is there anything you’d like to add?

I think one of the things we ought to be focusing on as we design the next generation of interaction technology is its potential for amplifying our expression. My working definition of the expressivity of a tool or instrument is how effectively it enables an idea to be made into a reality. The definition works as well for new interfaces as it does for musical instruments, and building tools that close the gap between the creative impulse and a realized instantiation of that impulse is my life’s work. Moreover, the delight that we feel when playing musical instruments is a target to strive for in the area of human-computer interfaces.

You’ve gotten a huge response from both the TED community and from the public. Is there something you’ve been hearing that you’d like to address or expand on?

I have heard so many people say: “My kids will love these. When can I get some?” The realization has been hitting us over the past few months that the potential for kid-oriented interactions is huge and meaningful. Someone else likened them to the Speak-N-Spell, but for the children of the 21st century. I like that comparison, since I have great memories of that device and I think its educational impact was important. If we can make a similar contribution I’ll be very happy.

Credit: TED.com
Watch David Merrill demo Siftables >>

New: Download this talk in high resolution (480p) >>

Watch Will Wright’s talk on TED.com, where you can download it, rate it, comment on it and find other talks and performances.

Read more about Will Wright on TED.com.