Marc Goodman studies how to prevent future crimes — not in the Minority Report sense, but by thinking about the ways that current and imagined technologies could be used to commit crimes. At TEDGlobal 2012 he gave a talk on some of the ways this could play out – showing examples of how drug cartels and terrorists could (and already do) use new technology to serve criminal purposes. While Goodman thinks (and worries) about what criminals will do with 3-D printing, flying drones and open access to our DNA, he also believes that the same open spirit could be used to help law enforcement keep up — and even crowdsource policing.
It was a talk that I found both fascinating and terrifying. I wanted to know more: How are these dangers avoidable? Should we sound the warning signal now, or are most of these hazards far-future? And can citizens be involved in policing without creating a police state? So I called Goodman in California to discuss.
If I’m a technologist developing some new wonderful tool, should I be worrying about the potential misuses? Or is it that, we have these technologies, and now it’s everyone’s job to worry about the potential?
I think it’s absolutely incumbent upon inventors to think about what it is that they’re inventing, because all inventions have unintended consequences. Splitting the atom was a perfect example. There were huge opportunities for energy. Many countries power large portions of their electrical grid as a result of nuclear energy. But the flip side of that is Hiroshima and Nagasaki. All technology is dual-use, going back to the time of fire. Fire could be used to warm yourselves at night, or to cook food that was previously impossible to eat. But it could also be used to burn down the next village.
I’m not suggesting that any specific technology be prohibited or banned. I absolutely don’t support that. I think technology should be developed and we should push our scientific and technological limits. But at the same time, inventors need to understand that they have a responsibility, and so they should consider what it is that they’re building and how it might be abused and how it could harm others. I’m not talking about a heavy government regulation, but I’m saying that inventors themselves have a moral responsibility. If their intent is to create something that could be of great benefit to people than they should go ahead and consider the potential downsides of the technology and try to innovate in a way that maximizes the upside and minimizes the downside.
Well, one thing we’ve learned from open and crowd-sourced innovation is that some people are very good at inventing things, but nobody can figure everything that a technology is going to be used for. So is there a way to get around that? If I’m running a company should I bring in somebody to think about it?
Absolutely. I teach at Singularity University, which is co-sponsored by Google, NASA, Autodesk and a number of other large, silicon-valley companies. And as an institution, our goal is to leverage technologies to positively impact the lives of one billion people over the next ten years. Every summer we’re bringing together students from 40 or 50 countries around the world. What we do is come up with clever ways to leverage artificial intelligence, nanotechnology, synthetic biology, robotics, big data, and social data to create abundance, to go ahead and solve some of the world’s most persistent and pressing problems regarding energy, access to clean water, food, global health and nutrition. I sit in the role of chair of Policy, Law and Ethics, which means that it’s my responsibility to generate a debate and a discussion on the public policy legal and ethical implications of these technologies.
We understand at our core the importance of discussing the ethical implications of robotics or nanotech or synthetic biology and then to consider the broader public policy perspective, and in some cases look at past legal decisions to see how that might impact our future. At the end of every summer our students are required to come up with a project to create either a company or a non-profit organization whose goal it is to drive public health, or to make an important contribution towards education, literacy and the like using exponential technologies. There’s a practical element there and we have discussions with our students all the time. When they’re talking about building, for example, a drone company that could deliver medicine in Africa in places where there are no roads, we talk to them so that they understand how their drones could possibly be compromised by warlords, and how they could be abused. And we come up with strategies for mitigation and encryption of the radio controlled networks to make sure that they’re used for the positive.
In my own personal life, I consult with any number of Silicon Valley start-ups, who by their nature are trying to drive and create wonderful abundance and good in the world. So I think it is a bit unfair to expect a synthetic biologist, somebody who is hacking DNA trying to find a cure for cancer to have a criminal mindset or a terrorist mindset to be able to think of every potential harm of their technology. That’s why partnerships are so important and critical to the whole process.
I am by no means an expert in all the underlying technologies. I mean, there are people that get PhDs at MIT and beyond that have spent their lives dedicated to robotics and will understand it at a level that I never will. But what I do have in my own experience is a fairly deep and extensive understanding of the criminal mind, and knowing how they innovate. Criminals and even terrorists are natural innovators. They have to exist outside their current legal regimes and systems of governance, so by their very nature they’re highly innovative. And through my professional work I’ve come to understand that sense of criminal mind and criminal innovations, and I share that with start-ups.
It sometimes feels to me like we get presented with an idea of two separate futures — one where technology is wonderful and makes everybody’s life better and another one where the world devolves into a post-apocalyptic mess. Talking to you, for the first time I’m seeing a vision of a world that’s much more in the middle, a world where things are generally getting better, but there’s all kinds of new problems. What’s the picture of the future you have?
I think both of those visions — the utopian future or the dystopian future — often come from science fiction. It’s going to be Star Trek, where we will all have a holodeck, and we’ll have total abundance and we can go to replicators for our foods and the United Federation of Planets will protect us all, or it’s Robo-Cop. And I think in terms of reality what will happen is it will be a little bit of both, as you said. I really am hopeful and I guess I’m not sure if that came through in my talk. I have spent a career in law enforcement. Probably my third sentence of my TEDtalk was that I’ve seen more violence and bad stuff in my life than most people ever will — and that’s informed my opinion of things. And I realize that many people haven’t seen that, which is great. So I understand that this can seem very frightening and very shocking. And of course, it depends where you are in the world, there’s some people in the developing world that have seen more pain and suffering than I ever will in my lifetime.
One of my favorite quotes, and it’s one that I strongly believe in, is one by the novelist William Gibson. In Neuromancer, Gibson says, “The future is already here. It’s just unevenly distributed.” Drug cartels are using robotic submarines to deliver narcotics. Criminal groups are already using robotics, criminals are using artificial intelligence when they’re scripting computer viruses. There are massive portions of our global internet compromised and botnet armies do exist. So the future is already here, whether or not people recognize the fact that there are people with ill intentions, people with intentions to massively harm others that are exploiting robotics, artificial intelligence and synthetic biology. That’s here.
How the future turns out is really up to us. I believe that. It’s not a role that we should leave to government. I mean, I’m an American citizen so I take the American approach to this, “We, the people.” We, the people hold the future in our own hands, and the future that we create is up to us. We have all the technological tools that we need to feed the world, to clean water, to move towards abundant energy and I believe that we will get there. I really, really do. And in every society throughout history, there have been people who, for their own personal reasons, have tried to subvert the greater good. We’ve seen this from criminals, we’ve seen this from so-called terrorists, and we’ve seen this from governments and dictators. So I think that that trajectory will continue, but I really believe in the power of technology to bring us closer to a more utopian version of the future than the dystopian. But understanding that one lone actor, with the access to the right technology, can drive much more harm is also increasing exponentially. And my main point is: That’s something we need to consider guarding against.
In your talk you discuss the 3-D printing of guns all the way up to genetically targeted bio-weapons. How far off are these? How speculative are these ideas?
Some of the ideas were off in the future, and some of them are already here. The focus of the talk was the future — not all of this has happened or is happening today.
To the specific two points that you raise: 3-D printing of guns, there is nothing to prevent somebody from doing that today. All the technology exists for that to happen. The thing that probably is preventing its widespread adoption at this point is the price point of a printer that does 3-D printing in metal.
Can 3-D printers print something that’s structurally sound enough to actually fire a bullet?
Oh, sure. In metal, that is.
[Note: A couple days after this conversation Goodman sent me this article, which shows partially successful test results from a printed gun. ]
And the genetic weapons?
In terms of printing or creating bio-weapons, that’s already here, that’s entirely do-able. The tools to do it are out there — the tools for the synthesis, the tools for the creation. Various ministries of health, most particularly the U.S. Department of Health and the National Institutes of Health specifically, have genetic databases where the government has already sequenced pathogens and have put the genetic code online so that researchers can create them and do great science.
I want to be very clear: For the overwhelming majority, biological research will inure to the public’s benefit. The fact that we will have people in their garages or in much smaller laboratories will be great, and we will see untold medical advances. But there are two potential dangers. People that haven’t been formally trained, particularly in bio-science safety, will make mistakes. And errors will occur, just as they do in traditional laboratories. Except there’s much less oversight around these issues, and so when those mistakes happen people can be hurt. Then there will be a much smaller subset yet which will go ahead and purposefully try to engineer bio-weapons writ large.
Aum Shinrikyo, the terrorist organization out of Japan that was responsible for the 1995 Sarin gas attack, tried to create a biological weapon. Going back to the mid-to-late ‘80s, Aum Shinrikyo spent over ten million dollars doing biological warfare research and they weren’t quite able to pull it off, which is why they opted for a chemical weapon.
Ten million dollars is not an insignificant amount. But we’re 20 years later and given that the advances in biological sciences and the democratization, the access of this information to the general public, has been phenomenal — off the charts, far out-pacing that of Moore’s Law. So, I believe that it would be trivial for somebody to do this today.
One thing I hear a lot is that most terrorist attacks fail. Is the argument that there are a small number of people who are capable of this, or is it actually a widespread?
Let me take the premise of your question and examine it for a second. You’re suggesting that most terrorist incidents fail. I don’t know where that thought comes from or how that’s backed up by research because it’s impossible to know the total sum of planned terrorist attacks. It’s not as if terrorists register in a database when they plan on blowing something up and then we can go back and compare it, you know, and say, “oh yes 42% have failed and 97% succeeded.”
So I have a little bit of a difficult time understanding the underlying assumption. I will say that there are terrorist attacks — plots — that are disrupted, and I would say there are others that are not disrupted. As to whether or not they all go to plan, I’m sure that they don’t, the same way that military operations don’t go to plan.
As to whether or not these will be isolated people or will this become a widespread thing, my impression is that it will be a small, small number of people that want to disrupt society and to fight against the injustices that they perceive by mass killing. Throughout history that has not been a large number of people, but when those incidents occur, they have significant impact.
The vast majority will be experimenting for phenomenal reasons, whether it’s to find cures or medicines for altruistic reasons or to re-engineer food, for the purposes of creating abundant energy. So I think that the vast majority of it will be good. The problem is, one person can create much more evil through these exponential technologies than previously possible.
Keep in mind that, professionally, this is what I do. I’m somebody who’s worked in law enforcement and counter-terrorism. It’s my job to protect the public and to think about these worst-case scenarios, because society would like to be protected from these things. There are professionals that look into this so that the general public doesn’t have to stay up at night thinking about it. I think that’s probably for the best. Though I believe that there is a phenomenal opportunity for the general public to help in protecting themselves and to become more involved in their own public safety as well.
Historically, if you look at terrorism compared to other sources of harm, from poverty to natural disasters, it affects a very small number of people. So the question was, how many resources should be devoted to fighting it if you have limited resources? What you’re saying is that we’re looking at a potential for much greater harm. This is a classic black swan problem, where you have some potential harm, but you don’t know how likely it is. How much do you devote to worrying about something like that?
What I focus on is the future of crime and terrorism. TEDtalks are narrowly focused — they can’t be all over the map. So for example, I was talking about various acts of crime and how criminal organizations are implementing technologies, and I talked about how narcos are using technologies to great effect. That was not a commentary on the root causes of crime. It wasn’t a commentary on global inequity in the world, or unequal distribution of resources. It wasn’t on the root cause of crime, it wasn’t on how to cure crime even. It was very purposefully focused, as many TED talks are, on the latest criminal and terrorist trade craft and how they’re using technologies to either commit more crime, better crime — crime at scale — or terrorist attacks and during these attacks to kill with greater lethality. I will also say I am not a psychologist or a sociologist or an anthropologist or even an economist. I’ve approached this as somebody who has spent a career in law enforcement. It’s important that be understood.
On resources, I believe that certainly it’s critical, particularly in these times of dwindling public resources, that we look very closely at how we allocate them. If you look at many countries around the world, certainly it’s true with the United States and parts of western Europe, the vast majority of the budget goes to national defense and public safety to the exclusion of other programs that may potentially impact crime and criminality — things like education, access to food, healthcare, sanitation and the like. And I get that. So how much should we spend? I can’t say specifically. I think there are many opportunities to spend more wisely. I also think that we spend money on many different things that are clearly ineffective.
I can’t say that we should have a 300% increase in what we spend on, you know, bio-weapon surveillance programs, but I think that we can spend more effectively and perhaps in many areas reduce and reallocate towards other non-defense, non-law enforcement functions. If you look at the very end of my talk, the important takeaway message is that the government can’t do this alone. By getting citizens involved in their own public safety, not only will we be significantly more effective, not only might the process be less intrusive, but it will surely be cheaper and better use of resources.
If you are getting people involved in their own security, how do you do that without the worry of creating a police-state, citizens-spying-on-each-other-type scenario?
Absolutely. It’s a question that I’ve thought about because we’ve seen many examples of this go awry, whether it was in the former Soviet Union or the Stasi in East Germany. Nobody wants to create a police state. This is not my area of expertise, so I want to be clear about that — but there are some really significant and draconian threats to privacy on the horizon. And if you think it’s bad now, it’s going to get much, much worse in the near future. There have been several TEDtalks about this by experts in the field so I will defer to them. I’ll just say that government is but one person capable of very significant surveillance. There was a story in today’s paper talking about a very large commercial network, with nearly one billion members, and the fact that it was doing facial recognition on everybody’s photos that they were uploading — creating the largest database of facial bio-metric data that has ever existed. That’s not a government database, that’s in the hands of a public corporation that’s not restricted in any way by any regulation in terms of how they might use that information. There are some very significant privacy concerns. I myself am concerned, and I’m at least as concerned about this data in the public sector as I am in the private sector.
Is there some way of forming citizen input to crime prevention that has a chance of avoiding these problems?
Certainly if you were to take it to the extreme, the creation of a vigilante police state is that slippery slope that we go down. I don’t think that has to be the case. If you look at sort of the original crowd-sourcing of crime prevention or crime investigation, it went back to medieval days where they would go out and look for people that had committed crimes against the crown. But more modern examples in the United States go back to the mid-1800s when posters of wanted people that had committed crimes were put up in post offices. So you would take a picture of John Dillinger and you would put it up in a post office and then people would look at the photo and say, “Oh, if you see that guy, please let us know.” This is not a commentary on whether any particular criminal is guilty and whether or not there should be a part of the state that focuses on crime in law enforcement. I understand that people, a large percentage of people, would rather focus on crime prevention and I’m all for that. But, that said, people do commit crimes. Some crimes certainly are subject to debate — there’s big debates about drugs and alcohol that are criminalized in certain countries. I’m not talking about that. I’m talking about violent sexual assaults, genocide, things of that nature. Even basic assaults, bullying of kids in high schools leading them to suicide. There are a lot of problems that we need to address.
So how do we do that without going to that extreme of the police state? What I would say is community policing and neighborhood watch are examples that have mostly worked fairly effectively. Which is, if there’s something untoward going on in a particular community, the community comes together and deals with it. So if a drug dealer opens up a crack house on a particular block, then people have come together. The good people, because most neighborhoods are populated by good people, and in fact they themselves are the ones that are most affected by crime. So when a drug dealer moves into a neighborhood, or a criminal organization opens up in a particular neighborhood, there’s then a battle that ensues between the police and the drug dealer that impacts negatively mostly the people that live in that neighborhood. There have been successful examples of community policing, where the police and the community partnered to set the standards of safety and security that they want in their communities. And while there’s certainly have been examples where that hasn’t worked perfectly, I think overall, having people participate in their communities — whether it be through police athletic league, football teams, baseball teams, community picnics, and the like — those are what build the bonds of our community, and I think that that’s a good thing.