Fellows Friday TED Fellows

It’s a bird! It’s a plane! It’s Fotokite! TED Fellow Sergei Lupashin brings aerial photography down to Earth

A Fotokite captured this stunning image of the stars and palms in the Okavango Delta in South Africa. Sergei Lupashin gave fellow TED Fellow Steve Boys one of the devices, to help him capture the landscape from unusual angles. Photo: Courtesy of Steven Botes

A stunning image of the stars in the Okavango Delta in South Africa. Sergei Lupashin gave his friend Steve Boyes, also a TED Fellow, a Fotokite to help him capture the landscape from unusual angles. Photo: Courtesy of Steven Boyes

With his tethered quadcopter Fotokite, Russian-born inventor Sergei Lupashin plans to put aerial photography and the power of unmanned aerial vehicles in the hands of of journalists, architects and artists. Interestingly, this device was actually inspired by a 2011 protest in Russia. We talked to Lupashin to find out more about this inciting moment, and about how he plans to push past the public’s fear of drones.

You’ve said that the idea for Fotokite came from witnessing a protest. Can you tell us more about that?

The protest I was referring to took place in Bolotnaya Square, in 2011, in response to massive irregularities in federal elections. Something very rare happened — a lot of people came out to protest in Moscow. For Russia, this was a huge event. But for whatever reason, the world media pretty much ignored it.

There happened to be a group of photographers nearby who usually do nature shots. They take quadcopters and octocopters to, say, the Sphinx or the Pyramids, and take these fantastic panoramas. They happened to be just around the corner, so they did a few panoramas of the protest. In a single image, you really got an the idea of the scale of this event. It was really eye-opening. Ironically, these photographers were completely apolitical. They were simply documenting what was going on near them. It struck me how powerful it is — how even a single photo from an aerial perspective can really change the world’s perception of situations and events.

This is still a motivator for us, and we’d like people to take the Fotokite to breaking news events. TED Senior Fellow Teru Kuwayama once said to me that you only need a very small shift in perspective to be able to make a great, unique photo. This is also quite interesting to me, because the Fotokite really opens up your envelope in terms of where you can place the camera.

You first came to TED as part of the team manning the Flying Machine Arena at TEDGlobal 2013. (Watch the talk, “The astounding athletic power of quadcopters.“) What’s the difference between the quadcopters you demonstrated there and the technology you use for Fotokite?

The quadcopters we showed at TEDGlobal were part a larger system. The Flying Machine Arena is essentially a motion-capture space configured for robotics, surrounded by a net cage. Motion capture is usually used in Hollywood to track actors, but in this case, we used it to track vehicles. The quadcopters are only able to fly so precisely because they’re being tracked by external cameras, in real time, with extraordinary accuracy. It’s a very expensive—a very specialized system that you couldn’t actually use in the real world beyond performances and installations.

So the question was: how do we build something for the real world, something really simple that you could use outside of a very controlled setup? The Flying Machine Arena served as an incredible sandbox for exploring various directions and concepts with the technology. We learned to build the Fotokite in this environment.

Every photographer knows how to use a tripod. So the idea emerged to build a tripod that could extend very high up, say, 100 meters. Initially, we were using different algorithms to make this “flying tripod” work, but ultimately, a tether proved to be a very elegant solution. The tether is always taut, so we can use it for two things. One is to interact with the vehicle — you can control it like a flying pet. The other is to let the vehicle know where it is, relative to the user. This replaces expensive cameras and GPS.

Lightweight and small, the Fotokite can be launched and ready for action in a  minute. Photo: Milan Rohrer/Fotokite

Lightweight and small, the Fotokite can be launched and ready for action in a minute. Photo: Milan Rohrer/Fotokite

How does it work? Is there a camera that reads where the tether is?

That’s the magic of it: there are no special sensors. It’s using the same sensors you have in your phone, for example. The magic happens is in the algorithms. So we’re using inertial sensors, very simple sensors that have been made affordable and incredibly reliable thanks to smartphones and consumer electronics, to measure accelerations and rates of rotation. Then we apply estimation algorithms to figure out the angle of the vehicle relative to the user.

So the sensors figure out the angle of the quadcopter to the person using the algorithms, with the tether as a reference point?

Exactly. There’s always a tether, and because it’s always kept taut, it’s always stretched, it’s always producing a force pulling on the quadcopter. We can observe this force using these sensors, and therefore, if we know what that force is, we can figure out where the quadcopter is relative to the person. This means we can also stabilize and do intelligent things based on that data.

There’s something really cool there as well. The quadcopter is aware of the user, of the person pulling on the leash. So we can actually use it as a communication channel. So you can do things like walk around with the leash, and the quadcopter moves with you. But you building on top of that, you can actually communicate with tugs and things such, so you can give it commands by physically pulling on the tether and it can even give you feedback by tugging back. It’s as if it’s a flying dog.

How high can it go, so far?

We’ve gotten up to 150 meters, which is when we ran out of string. We actually use dog leashes, for convenience.

I think that the main idea is to not go very high, because you want to be close enough to what you’re filming to actually capture detail. What’s been interesting so far is the 0 to 30 meter range. You can imagine if you’re a journalist or an architect or archaeologist, you always have a particular scene that you’re filming. When you want that, you need perspective.

In our latest version, we also offer a special leash that feeds power up to the Fotokite from the ground, which lets you fly as long as you can provide electricity from a backpack battery — from your car, from the grid, and so on.

Above: See what the Fotokite can capture from above in this demo video featuring clips from test flights — from Tokyo to Burning Man to the Okavango Delta.

What kind of camera is on there?

Right now we usually use a GoPro camera. It’s very lightweight, which means the whole thing is safer and more comfortable to use. We support other, heavier cameras as well, but the payload could be really anything — an air-quality sensor, a radiation sensor, and so on. It’s a platform for carrying anything, which is really neat. For example, we’re doing a research project where we’re trying to find baby deer using infrared cameras. We have this payload bay, and you can 3D print any sort of holder for your instrument on your printer. It’s pretty straightforward.

Can you talk about the ethics of drones? People are concerned — how are you addressing that?

With drones, there’s a lot of concern around privacy. People are afraid that someone can use this thing to observe them — whether it’s the government, or other people taking pictures of them in their garden, on the beach, wherever. At the same time, there’s a clear benefit to having this perspective. For example, any time you’d normally need to climb up high just to look at something — like to take a photo of your property, or inspect the roof — you could use one of these remote cameras. So there’s a conflict in terms of how allowing one set of people to pursue this unique perspective easily infringes and intrudes on the privacy and comfort of others.

This is actually a pretty old problem. Aerial photography can be traced back to at least 1906. There’s a beautiful photo of San Francisco after the great earthquake taken by George Lawrence using kites — a beautiful, huge, high-quality photograph. If you look at the legal history as well, it goes back decades.

With the Fotokite, the difference is that there is a physical link to the operator. This means that  anyone can observe who is responsible at any point. Hence, a lot of these ambiguities get eliminated. You always know who is operating the vehicle, and you always know if you’re safe or not. In other words, is the machine physically capable of reaching you?

Do you think Fotokites will need to be licensed?

What we’re seeing is in a lot of countries, you do have to have a license to operate a drone with a camera, for commercial use. In the US, actually, they tried to ban drones — any sort of unmanned aerial vehicles (UAV) — for commercial use it altogether. I think they made one exception, which is a large airplane, just now. However the FAA in the US (and some other authorities around the globe) typically exclude tethered vehicles from regulations and/or certification.

But how is that different from, say, Google going around with a van and photographing every single street window?

With a normal, untethred UAV, you have no idea who’s operating it. So it is quite different. With a car, you can always say there’s a driver who’s ultimately responsible. With a UAV hovering 100 meters away, you don’t know. It could be anyone within a kilometer.

In the US, I think the main concern is not actually privacy, but airspace regulations — making sure that these things don’t fly into airports or into other airplanes. But for example in Switzerland, for Google Street View, they had to lower the height of the cameras to around two meters, because they were considered too intrusive. Ironically enough, in Switzerland, you can actually fly a drone as long as it’s line-of-sight, and you have insurance, things like that. In the US, as long as you’re doing commercial work, you can’t.

Right now, the world is in this very funny transition phase where all of the rules are very different from place to place, and they’re changing very quickly. We have to consider all of this as we develop.

The Fotokite aloft. Photo: Milan Rohrer/Fotokite

The Fotokite aloft. Photo: Milan Rohrer/Fotokite

Where are you in development now?

We’re doing some early trials with first customers, especially journalists, and learning a lot from that. We’re always improving the robustness and practicality of the device. The latest version works with your phone for live preview and control, for instance. There’s a lot of footwork that goes between having the proof of concept — what I demoed at TED — and actually having even a minimal, basic product.

Have you had a lot of interest in Fotokite since TED?

Yes. TED was really great in terms of helping us define the vision and the mission of the company. It’s really motivating day-to-day for everyone in the office to know that we’re doing this for a decent reason. And working with other TED Fellows has been really cool. For example, we’re sending Fotokites out with TED Fellow Steve Boyes to the Okavango Delta in South Africa, to help him document the expedition. This delta is one of the richest wildlife areas on the planet and Boyes is currently out with a team of scientists, artists and engineers, collecting data in the area and, essent taking its pulse w.

The BBC also used a Fotokite unit to film part of a show. This was a real learning experience, because they weren’t interested in any of the fancy features that we were talking about. For them, they are in the field, it’s rainy, and there’s a group of five people filming badgers or something. They’re filming from 6am to midnight — they’re tired. It’s terrible weather, everyone’s grumpy. So they loved the idea that it’s reliable and super easy to use. And that it’s compact.

Are you interested in photography, or art? What was your path here?

I spent a lot of time in academia on various robotics projects. I worked on an autonomous car for the DARPA Urban Challenge. I came to Switzerland to work on robotics from an academic perspective, basically building cool toys. And I think that Fotokite reflects a bit of my feeling that I was getting quite isolated from having a real impact on the world. It’s cool when you make a really nice YouTube video where a quadcopter plays a piano, and it’s fun — but I wanted to make something that would be really useful in normal life.

At the same time, I saw a lot of people building flying cameras from hobbyist components. This is kind of also leading to a weird transition market where you really have these really large things flying in the air, and they’re actually quite dangerous. I mean, sometimes it’s done professionally, but sometimes it’s also just cobbled together. So if we can build something a bit more elegant that does the same job, it would be a nice contribution.

Lupashin's Fotokite selfie with Edward Snowden via Beam, the self-propelled telepresence robot that allowed Snowden to speak onstage at TED2014, Vancouver.

Sergei Lupashin takes a Fotokite selfie with Edward Snowden via iBeam, the self-propelled telepresence robot that allowed Snowden to speak onstage at TED2014. Photo: Sergei Lupashin

Where are you customers? Who are they?

Our customers are really all over the world. This is really cool, because we get to talk to a lot of different people, who have very different concerns, and very different motivations for wanting Fotokite. And because we’re actually building the solution, we have a hand in the conversation as well. We’re not passively providing something; we are trying to encourage uses of the technology that we think are responsible. For example, journalism and news gathering is a particularly interesting application looking for a practical solution — drones are used a bit today, but they come with a swarm of complications like certification, pilot training, safety concerns, accountability and privacy issues.

But you can’t control who will buy your product, can you?

In the end, of course not. But we can certainly control what kind of projects we do, where we put our focus. For example, we’re working on making cameras stable and usable for newsgathering by providing appropriate live-streaming technology packaged in the unit. We could just as easily be working on an infrared camera that has 100x zoom, that can follow someone — clearly a much more suspicious project. We choose to pursue only what we believe can be shown to have clearly legitimate, public-interest uses.

What’s the ultimate vision? Will Fotokite be on shelves at a department store? 

We are considering direct sales, also a Kickstarter campaign. We’re trying to keep the perspective that what we’re building should be a modular platform: so a consumer Fotokite should hackable and compatible with off-the-shelf, open-source autopilot. So people could add their own modifications and think of new crazy use cases, like flying umbrellas. Or if you have an idea that you want to make a lamp out of this thing, or a Burning Man art installation, you can do that without reverse-engineering everything.

Were you expecting to be an entrepreneur?

Not really. But I also didn’t expect to earn a PhD, so I guess things develop with opportunities. It’s been a wonderful experience. At the end of the day, it’s about learning new things, and also trying to make a difference. That’s the whole thing. Starting a company — and building a product that empowers people like journalists in what they do best — is a really great way to do it.

Above: Steve Boyes shares some preliminary footage of the Okavango Delta captured using the Fotokite.