When AI Takes Over

He spoke with Vision publisher David Hulme about the related challenges and risks.

DH One of your colleagues, Nick Bostrom, has commented, “The transition to the machine-intelligence era looks like a momentous event and one I think associated with significant existential risk.” If so, what is the risk, and is it existential?

AS One can look at the transition from prehuman to human intelligence and see the consequences for the world. Right now we’re controlling all the ecosystems on the planet, sometimes doing a very bad job of it. Our fellow apes are actually at our mercy, and we are transforming the climate in ways they can’t even comprehend. So if we were to get machine intelligence on par with human intelligence, we might get a similarly dramatic transition of a situation on earth, which might of course not be to our human favor.

DH Stephen Hawking and colleagues have written, “Success in creating AI would be the biggest event in human history. Unfortunately, it might also be the last, unless we learn how to avoid the risks.”

AS This goes back to a classic paper by the statistician and computer scientist I.J. Good in the 1960s, called “Speculations Concerning the First Ultraintelligent Machine,” where he suggested that we could make machines that make better machines. And then he noted that this would be the last invention we would ever need, because then the machines could invent whatever we needed. But he also added, prophetically, “provided that the machine is docile enough to tell us how to keep it under control.” The problem, of course, is controlling something smarter than you, or even telling it what to do in a way that can’t be misinterpreted or doesn’t turn out to have bad consequences. That is very tricky.

DH At what point in the development of an AI application should designers consider the malicious applications of their research?

ASWhen you try to develop an AI application—or for that matter, any application—you should probably give a long, hard thought to “How can this be misused? How can people with bad intentions or stupid intentions misuse it?” Or even, “How can intentions emerge accidentally that lead to bad consequences?” And you need to start very early, because, just like security, this is not something you can add on later. Once you’ve done the core of your program, it’s going to be very hard to fix.

DHWhat keeps machines from actually taking over?

ASRight now, machines don’t take over the world in the normal movie sense, because they don’t have any agency; they don’t plan and perform actions. But in a sense, of course, machines have taken over the world in that we’re very dependent on them, and we can’t actually avoid it. In the long run, we should expect machines to have much more agency and much more ability to change the world. At that point they might accidentally take over, or just because somebody gave them an order that they interpret in a sense that practically leads them to take over the world.

DHSo if something went haywire, like some sort of AI decision to shut down the power grid to preserve its own integrity, will humans still have the power to override a bad computer decision?

ASNormally people say that if a machine’s misbehaving we can just pull the plug. But have you tried pulling the plug on the Internet? Have you tried pulling the plug on the stock market? There are many machines that we already have where it’s infeasible to pull the plug, because they’re too distributed or too essential.

“It might very well be that the forms of machine intelligence that we decide to use are going to be so integrated that we can’t pull the plug.”

We can still have other ways of controlling them. Again, this is a matter of design. A sensible designer makes a system that is failsafe; when something goes wrong, you can shut it down carefully, or it stops doing whatever made it go haywire. But that requires careful design. And as we know from software, a lot of us humans are not very careful designers.

DHOne of the dangers inherent in the application of AI to decision-making is that, while it may be objective, it may not be capable of fine discrimination or nuance—say, in deciding whether a person should go to jail. In that case, would AI be capable of extending mercy or empathy?

ASNormally when we speak of mercy, we mean a relatively rich concept of mercy—of actually understanding somebody, having enough empathy to be in that person’s shoes, and then deciding, “I’m going to forgive this person because of their situation, because of their role.”

That might be very hard to do in AI. I think it’s logically possible to do. I actually do not think there’s anything a human mind can do that a machine cannot do. However, that’s not going to be easy to do. It’s much easier to have a thinner concept of mercy, where maybe the machine is doing good pattern matching and imitating human patterns of behavior that we would call merciful, without actually having any content.

Now, in some legal situations, that might be alright. We might say that sometimes an objective decision, even if it has some biases, might be better than the less objective decision a human judge might make. But in many situations we would say that the humanness of a judge or the humanness of a jury actually matters, even if they’re not intellectually as capable as a machine.

DHWhen do you foresee artificial general intelligence becoming a reality, and are you concerned about the possibility?

ASPeople have been predicting “artificial general intelligence in maybe 20 to 30 years” for a lot of years. And most of the past predictions, of course, haven’t come true because we humans are very bad at actually predicting things like that. Given current radical progress in machine learning, things are looking up, but we’re still extremely uncertain what it would actually take to make a real artificial intelligence. My own personal view is that we should have an extremely broad probability distribution. It might happen in a decade; it might take many decades; maybe it takes hundreds of years.

But from a safety standpoint, we shouldn’t be calmly assuming that it’s going to be far, far in the future. We should assume that it’s going to happen relatively quickly and want to make safeguards before that, because general intelligence is much more powerful than narrow intelligence. Narrow intelligence might be disruptive enough to our society and economy, our ways of making war or solving problems; but general intelligence can be amplified and can solve problems we couldn’t even conceive of. That makes it tremendously important to be careful when we develop it.

But preparing for something that is unforeseeable is, of course, very hard.