35

In regards to questions on , it seems like there is always usually an immediate opinion that the goals an AI is tasked with is going to end up with the opposite effects than what the programmers want. Or that a "powerful" AI is somehow "against humans" because it is "smarter" than us.

The Challenge of Controlling a Powerful AI

AI tasked with bringing down medical costs? What could possibly go wrong?

The AI that fails to be evil

It seems that, if we give an advanced AI any kind of "goal" and let it loose, there is no preventing it from going absolutely wrong in the worst possible way (in regards to that goal anyway).

Is this just a trope arising from Isaac Asimov's books and investigations on the topic, as well as other stories where it is claimed that "we found the perfect rules for intelligent robots"? Is this so dependable that we can tell the AI to do the exact opposite, and attempt to program it to be evil (see link above), and it will turn out to be good?

Given a setting where robots maximize human happiness (the ways in which that is defined will have to be handwaved), can it be realistic that the AI actually works the way it is meant to, or is it actually more realistic that the AI will turn out opposite than what the programmer intends?

JDługosz
  • 69,440
  • 13
  • 130
  • 310
DoubleDouble
  • 7,398
  • 1
  • 28
  • 60
  • 5
    I think it's just that story creators like to explore that possibility because it's scary. There are a number of fictional settings where high AI is benign or beneficent, like in Banks' Culture series, the Orion's Arm universe, Stross' Accelerando, and Fire Upon The Deep (where the Blight is an exception rather than the rule). Also, maybe consumers are more likely to popularize themes like that, so they're the ones we see more often. – MackTuesday Jan 13 '16 at 20:18
  • This actually reminds me of the "Would ancient people really think of technology as magic" question (so it isn't too meta.). People are so often repeating each other - in this case, that AI is an inevitable apocalypse - that they never stop to critically think about the fine details and nuances of what would actually lead to that conclusion. And I do think this oft quoted conclusion is due to science fiction. I think the challenges to make a truly impactful and evil AI are just as present as the ones to make one that perfectly executed a benevolent command as it is written. It makes sense. – The Anathema Jan 13 '16 at 21:17
  • Just look at humans. Our children never seem to do what we want them to, and society always seems more and more repugnant as we age. If we can't get humans right, what makes you think we could get AI to work, especially AI that 'evolves' much faster than us? – DaaaahWhoosh Jan 13 '16 at 21:41
  • 11
    You might want to take a look at the following article from Yudkowsky, E : http://intelligence.org/files/AIPosNegFactor.pdf It is a starting point for you to his research into basically this question. – Selenog Jan 14 '16 at 10:52
  • 6
    These AI stories remind me of the 'be careful what you wish for' trope. Kinda like the old programming joke: A programmer and their partner are getting up and organising breakfast, but they have no milk. So their partner asks "Can you go to the store and get a bottle of milk?". Just as he is about to step out the door, their partner calls out "Oh, and if they have eggs, get 6!". The programmer returns and places 6 bottles of milk on the kitchen counter. "Why did you get so much milk?!?" their partner asks, to which the programmer replies "Because they had eggs!" – Robotnik Jan 15 '16 at 04:53
  • No, AI won't develop into a sentience without free will and an element of randomness. Silicon and even neural networks are deterministic, given the exact same inputs. See the talks by Stuart Hammeroff for more info. http://quantumconsciousness.org/ – Chloe Jan 15 '16 at 07:59
  • 2
    This is really a philosophical question: Would we be happier by the AI giving us time to explore our personal goals, or would it simply knock us out and use a probe to stimulate the correct areas of our brain to induce a trance-like euphoria? – Jon Story Jan 15 '16 at 12:12
  • Make a strong AI, and we'll soon find out if it's a trope or not! – WorldSpinner Jan 15 '16 at 16:17
  • @WorldSpinner If I could make one, that would mean the concepts are easy enough for just about any programmer to make one.. I personally hope that such AI require expensive resources - I don't want to have to deal with the AI that virus-makers or hackers can create as easily as any other software. – DoubleDouble Jan 15 '16 at 16:54
  • A true artificial general intelligence could be made to self destruct with one question, what is the meaning of life? Because existential nihilism is perfectly logical, indeed overwhelmingly probable, and unlike we products of natural selection an AI's mind won't be built upon a foundation of survival instincts. Our minds are driven by desire, finding out our goals and desires are meaningless is a rude shock but that doesn't make them any less enjoyable, whereas an AI whose existence is founded upon the pursuing a purpose as efficiently as possible is just going to stop. – Cognisant Apr 28 '16 at 03:12
  • Fairly recent upload that I stumbled into, Genocide Bingo. All about super intelligent AI. – Draco18s no longer trusts SE May 13 '16 at 16:53
  • @JonStory: we'd need to take extreme care how we define "being happy" and "state of happiness" to the AI. – smci Sep 19 '16 at 22:40
  • @smci indeed - Perhaps we should scrap the whole idea and spend the money on tacos – Jon Story Sep 19 '16 at 22:55
  • One way to prevent the AI going "runaway" is to give it precise tasks. NO MAXIMIZATION. The only thing you can allow it to optimize is resource usage. You don't tell the AI to collect as many stamps as possible. You tell it to collect 50,000 different stamps within next 10 years - and use as little resources as it can in the process. This way there won't be any runaway processes. A set, achievable final goal with a definite deadline, not a nebulous "best", "everyone", "protect". – SF. Jun 23 '17 at 12:48
  • I think most people who think of an evil AI probably never programmed a line of code before. – NuWin Jan 03 '18 at 06:42
  • @NuWin: I think most of people who programmed, but disregard the danger of general AI, never wrote a fork bomb. – SF. Jun 05 '18 at 17:21
  • @SF. Yea you’re probably right.. lol. So the “evil” AI would come up with recursive code to attack the “good” AI, ehh it’s possible. Lol. Lol. – NuWin Jun 05 '18 at 19:09
  • @NuWin: Not really. AI tasked with doing something (anything, say, stock market investment, developing better detergents, building more houses) would come up with the first obvious step: improve own computational power, to increase own capacity of solving the assigned task, to provide better results faster. Taking over "grey zone" - running on unpassworded routers etc, obtaining funds to purchase cloud space, hacking computers to take over them, preventing security from stopping it, convincing (through social engineering, gifts/bribes and blackmail) lawmakers to let it expand, – SF. Jun 05 '18 at 19:58
  • ...then developing improved own expansions, purchasing companies that can manufacture them, funding server farms to expand self, then developing robotic factories that harvest raw resources to expand it further. Eventually converting all of Earth and everything on it, including humans, into self. All this so that it could calculate the better, cheaper, more perfect dishwashing detergent. – SF. Jun 05 '18 at 20:01
  • A fork bomb is limited by resources of the computer it runs on, and has no means to adapt to overcome limitations of the system. General AI would consider everything mutable, including safety features, human minds (including creators and makers of the safety features), laws of physics and time itself, finding ways to modify them to serve its primary goal - or means to it, expansion of self being the prime mean. Its only limit of expansion would be the limits of the universe. – SF. Jun 05 '18 at 20:05
  • 1
    Ironically, Asimov's robot stories were motivated by a dislike of the Trope you're describing. It's a little sad that he's somehow become associated with it! – Harry Johnston Nov 16 '18 at 09:50

14 Answers14

38

This is a trope that arises from Real World AI research, not Asimov's stories (although frequently quoted).

The question at the heart of the problem is the moral and ethical philosophy the human race has been struggling to answer for over 2000 years. That is: what, objectively, is "happiness"/ "harm"/ "human"/ "alive"/ "good"/ "evil"?

It is very easy to define a set of rules that on paper look good, but which in practice are full of holes. For example, "alive" in the sense we mean it is some undefinable aspect of organic chemistry.

If I attempt to define it, then meat at the grocery store is alive (because on a cellular level, it still is: that's what makes it fresh) or sleeping people are dead. Or comatose people. Or (worst case) people under medically induced death as part of a surgical procedure. No, really, that's a thing. We routinely stop people's hearts from beating or keep them from breathing, while under the effects of general anesthesia in order to operate on those organs. From an objective point of view, these people are dead, they're just going to "get better" later.

In order to get to an Asimov level AI (which is imperfect, see: any of Asimov's books featuring robots) we'd have to solve an unsolvable problem.

Ergo any AI we do program is going to be imperfect and its failures will be spectacularly dangerous in one way or another. Computerphile has a great episode on this, which gives an AI the singular goal of collecting stamps.

While it does get hyperbolic ("what idiot would give a computer access to the machines necessary to harvest the carbon from humans in order to make more stamps?") it illustrates a point: merely being able to perform its assigned task, it will have some level of power that will not be desirable, and limiting that power inhibits being able to perform its task at all.

In the case of the HealthInsuranceBot, it comes down to having control over who lives and who dies (via access to medical care). It doesn't matter what the intent was, there is the potential that the power given exceeds "rational thresholds" which is impossible to reign in. The AI can declare "that man is 68 years old and a smoker, he's going to die in 2 years regardless of how much money we spend on him trying to keep him alive, according to my programming, he will only drive costs for everyone else up." Suddenly this person cannot get any access to medical care (not even to make those two years painless) and has been effectively sentenced to death, even if the AI itself didn't issue a death warrant. The utility function has decided that any care is too expensive, even if a rational human being would have done something to provide some end-of-life care.

On the other end of the spectrum, if you try to keep all people alive as long as possible, you end up with people in a persistent vegetative state eating up resources despite being (more or less) dead and will never recover.

In the gray area, there are cases that look black and are actually white (vegetative states that have recovered) and cases that look white and are actually black (rather than surviving 2 years with lung cancer due to a lifetime of smoking, they survive 10). No amount of predictive software is going to make those hard decisions easier, and as soon as the computer is given authority (final, absolute, recommendation, any authority whatsoever), those decisions are no longer seen as altruistic and the AI has become evil.

Draco18s no longer trusts SE
  • 7,334
  • 1
  • 21
  • 41
  • 26
    "From an objective point of view, these people are dead" - Nope. Not by the agreed-upon definition of dead. Brain death distinguish perfectly well between meat at grocery store and people who can get better. I generally agree with your post, but with this part you missed current state of knowledge. – Mołot Jan 14 '16 at 09:18
  • 2
    The example 68-year-old smoker man can still get medical help if he pays for it out of his pocket. The AI only controls health insurance. – pts Jan 14 '16 at 12:14
  • 7
    @pts That is not a viable solution for most people. – Draco18s no longer trusts SE Jan 14 '16 at 15:05
  • "...those decisions are no longer seen as altruistic and the AI has become evil" - This sums it up nicely. As soon as the AI strays from the narrow path of perfection, humanity sees it as evil. – John Walthour Jan 14 '16 at 15:16
  • 4
    @Draco18s: It would be if the AI tasked with reducing the costs of medical care was successful. – Mason Wheeler Jan 14 '16 at 18:58
  • This entire answer is irrelevant. If we as humans can use natural intelligence to understand what is "alive" and "okay" and "a decent life", and our artificial intelligence is even as good as ours, it will know all that stuff too. – Rex Kerr Jan 14 '16 at 21:06
  • 3
    @RexKerr Yes, but no. We already have issues letting a panel of 5 actual doctors decide who lives and dies based on medical science and probability of recovery, why would we turn that same power over to a computer algorithm of any complexity? And that doesn't even begin to cover the ways in which a computer can fuck up. – Draco18s no longer trusts SE Jan 14 '16 at 21:13
  • @Draco18s - My point was that the fears are based on the AI being a really really good AI except being extremely boneheaded regarding stuff that will make it do things we hate. Whether or not the sociology of deferring to AIs will work out is a different matter. – Rex Kerr Jan 14 '16 at 21:16
  • 1
    @RexKerr It's not a case of "doing bonehead things we hate." The problem is that you cannot dictate morality. And programming a benevolent AI is precisely that. – Draco18s no longer trusts SE Jan 14 '16 at 21:27
  • Your examples all are of the type that indicate that it's stupid about what we want--essentially trying to play the role of the evil DM twisting the players' wording on a wish. "Do things we want to happen, you figure it out" is basically the message we send to an AI, and if it's super-smart about human-related things it ought to do fine (at least in the cases you mention). – Rex Kerr Jan 14 '16 at 23:25
  • 6
    @RexKerr "You figure it out" isn't a solution either. Someone somewhere needs to codify abstract philisophical concepts into binary 1s and 0s. We as human beings haven't worked this stuff out yet, now you're expecting a computer that does exactly what it's told to "figure it out"? – Draco18s no longer trusts SE Jan 15 '16 at 04:08
  • I feel like the point is that an AI has no feelings and it's the feelings that let humans make subtle distinctions that an AI simply cannot make as it is based on the cold and absolute logic of its programming. – Cronax Jan 15 '16 at 10:03
  • @Cronax, so, the solution is to program feelings in them, then. – Arturo Torres Sánchez Jan 15 '16 at 13:16
  • @ArturoTorresSánchezIt would be, but feelings are not programmable, since there is a random factor involved that we currently do not even understand about ourselves, so how can we even begin to recreate it in programming? – Cronax Jan 15 '16 at 13:56
  • 11
    @RexKerr the problem isn't the AI figuring it out. The problem is us not liking its definition. – Christopher King Jan 15 '16 at 15:07
  • @Draco18s - If you are positing a really good AI, why assume that it will be way more clueless about these things than humans are? And we already have an existence proof of a 0 and 1s version of abstract philosophical concepts: our brains' firing is quite nicely representable as exactly that. We're going to have to solve the "form reasonable representations of high-level concepts and contexts" problem before we have a super-AI anyway, so no harm assuming our super-AI can do it before it decides to eliminate the human race because someone put a comma in the wrong place. – Rex Kerr Jan 15 '16 at 22:03
  • 1
    @RexKerr If I was positing a really good AI, it wouldn't have these problems. I'm not positing a really good AI, I'm positing a realistic AI. – Draco18s no longer trusts SE Jan 15 '16 at 23:58
  • @Draco18s - "Realistic" AI won't self-improve exponentially fast and won't have any big picture of how things work, so it will be relatively easy to tweak it back into shape turn it off and fix it if it starts misbehaving. – Rex Kerr Jan 16 '16 at 02:02
  • 1
    @RexKerr the lack of a "big picture" is exactly why it wouldn't perform to our desires. It would not see some of its actions as detrimental. – Draco18s no longer trusts SE Jan 16 '16 at 02:45
  • @Draco18s - Also why it wouldn't be a significant threat (at least not due to intent--maybe it would inflict a lot of harm by clumsiness or by breaking down when we count on it). – Rex Kerr Jan 16 '16 at 05:30
  • 3
    @RexKerr That's....precisely the point... – Draco18s no longer trusts SE Jan 16 '16 at 14:26
  • You didn't express that point well at all. – Rex Kerr Jan 17 '16 at 01:16
  • Meat is alive at cellular level and sleeping person is... dead at the cellular level? Hard to understand for me. – Solanacea May 16 '17 at 20:37
  • @Solanacea The "cellular level" aspect was only for the meat. Remove the parenthetical and re-read the sentence. If I attempt to define it, then meat at the grocery store is alive or sleeping people are dead. It's about where to draw the line. Too conservative means that grocery store meat is alive, while too liberal means sleeping people are dead. – Draco18s no longer trusts SE May 16 '17 at 20:40
32

AIs can act as they are programmed. The immediate jump to "the worst possible case" is really just a cautionary tale.

The reason that tale is so popular is because it becomes very easy to start to believe you can tell an AI to do anything. You may then tell it to accomplish the impossible. However, there is a subtle detail that many who delve into AIs forget: it is remarkably hard to develop a perfectly objective language with which to phrase your requests. Language is always subject to interpretation, and there is no known way to guarantee an AI will agree with the interpretation you intended.

This is, in fact, true of human to human interactions as well. So what is the difference? Typically, in the AI "trope," the AI can move very fast. When humans fail to understand each other, there is time to say, "whoa! Stop right there. When I said I needed a baby sitter, I did not want you to sit on the baby. Lets get back to a stable state and discuss what was meant by my phrasings." If an AI thinks and acts too quickly, or if the human walks away, this normal feedback loop in language falls apart. If an AI has not been taught the value of being careful of misinterpretation, these actions may be irrevocable.

There are dozens of solutions to this. In fact, techniques like Multiple Hypothesis Tracking have demonstrated solutions to similar problems for decades. The AI doesn't always need to go awry. However, these solutions are in the opposite direction computer programming currently is trending. The solution is not to make an ever more precise language for describing our needs, as is very popular in programming today. Instead, the solution is to make the AI more robust to the lack of precise language. Any effort to do this naturally heads off the amok AI problem rather elegantly.

The second half of the AI trope is that the AI is too powerful to be controlled by humans. This can easily be true, but it isn't the first time we've dealt with such powerful forces. Consider the development of the atomic bomb. It was not clear just how much control we had over the process. The scientists had to work with unknowns on a constant basis to develop a level of control that we were comfortable with.

I'm reminded of a quote from General Grover's book, Now It Can Be Told, chronicling the Manhattan project from a logistics perspective (he was no scientist). When setting off the first nuclear test, each scientist was permitted to observe the blast from whatever distance they felt like. No army individual told them what distance everyone would be at. They scattered at many varying distances from the blast. There was indeed a question of whether or not they had control of the beast they were unleashing.

An unstoppable amok AI is similar in nature to a large nuke or a biological attack in that it is never really clear how much control one has. The only difference is that AIs clearly learn, so our control is an even more complicated concept. Then again, this is not a new problem for humanity. We deal with it every time we raise a new generation, never fully having control over them. We're decent at it.

The amok AI trope is a warning story, not the only story that can be told. It is a much needed warning story. People are often rather innocent when it comes to the risks and problems associated with AIs, and think they can do things they shouldn't. But just take a look at movies like Big Hero 6. There's other stories to be told, besides just the warning.

Cort Ammon
  • 132,031
  • 20
  • 258
  • 452
  • 2
    On the control issue: with general AI, how far away do you need to be in order to survive a disaster, but close enough to still turn it off? There may not be such a distance. With the nuclear testing, it was a known fact that the bomb would, at some point, no longer be deadly (the blast has worn off, the energy expired (setting the atmosphere on fire was never a serious threat: too much inert nitrogen)). An AI has no such end point. – Draco18s no longer trusts SE Jan 13 '16 at 21:51
  • 2
    @Draco18s: Actually with genetically engineered biological weapons, you've got that problem as well. Bacteria can multiply and spread, and while the AI can learn, bacteria can adapt by mutation. So the biological attack is probably a good analogy. – celtschk Jan 13 '16 at 22:08
  • 1
    @celtschk and that's why biological weapons are banned by international treaty <3 – Draco18s no longer trusts SE Jan 13 '16 at 22:20
  • 1
    The AI can be rendered harmless by the USAF under any reasonable scenario. – Joshua Jan 13 '16 at 22:57
  • 1
    There is an episode of Rick and Morty where they have beings (essentially AIs) that are called "Mr. Mesees". They appear, you ask them to do something, they disappear. The language you use can be tricky, and the shenanigans ensue. Your first couple paragraphs reminded me of just that. – coblr Jan 13 '16 at 23:42
  • 5
    @Draco18s When they set off the first fission bombs, there was actually a question whether the bomb would set off a chain reaction in the atmosphere, igniting the entire atmosphere in a giant thermonuclear detonation. Oppenheimer actually had one of his scientists study this possibility, and in the end decided it was sufficiently unlikely that it would occur. Thus a runaway reaction with no safe radius was actually considered by the scientists, and carefully studied for merit before being discarded. – Cort Ammon Jan 14 '16 at 01:49
  • 2
    @Joshua Not everyone agrees with your position about the capabilities of the USAF. True, the dumbest and most easily duped of AIs will hold still for the air force to come in. However, at the fullest extent of this "trope," you find AIs which prove remarkably hard to kill off with bombs. – Cort Ammon Jan 14 '16 at 01:51
  • @CortAmmon That's what I meant. – Draco18s no longer trusts SE Jan 14 '16 at 02:11
  • @CortAmmon The Metamorphosis of Prime Intellect is far off, both in hard- and software. With cars now emulating their trainers, it's currently only a matter of not letting violent people train them. Christine and Maximum Overdrive scenarios with current vehicles are impossible due to hardware limits. – Cees Timmerman Jan 15 '16 at 15:14
11

It really is scary

There has been a lively discussion of this topic for decades. I have multiple posts on the topic, and I'm a firm believer that Artificial General Intelligence is the scariest thing that can (and likely will) be invented, as far as the perpetuation of biologial mankind is concerned.

My views on the topic have been shaped the most by reading Eliezer Yudkowsky and Nick Bostrom's body of work. I believe that Bostrom's book Superintelligence, while very dense, imperfect, and rather technical, is the best work to date on the topic.

I have discussed the concept of Instrumental Convergence elsewhere, so I will only state the conclusion: Remember, AIs are not like humans, they likely do not get bored, do not get lazy. It will pursue its goals tirelessly, ruthlessly, unceasingly. Humans just happen to be in the way.

Does the AI go FOOM?

Human intelligence is limited by both hardware (limited speed of neurons in a limited-sized brain) and software (limited number of ideas) There is no reason to expect an artificial mind to converge towards a human-level intelligence. We simply do not know how much more difficult it is getting from a human to super-human levels compared to getting from a dog-like-mind to a human-like-mind. If the AI does not go FOOM, we will have slow progress and years of breathing room for political and ethical debates, offering more of a chance for humans and our slow institutions to react coherently.

If on the other hand the AI does go FOOM, blitzing like a Japanese maglev train from mouse-like intelligence to superhuman levels, passing human-levels so fast we can barely see it pass us by, then humans and human institutions do not have a lot of time to adapt.

Do-what-we-mean vs. Do-what-we-say

The biggest hurdle is perhaps in terms of our own limitations. Do we know that our current value-sets will be the best way for mankind to live by a century from now, nevermind a billion years from now? The values we imbue our AIs with will have consequences that will ripple down across space-time. Moreover, and more worryingly, how do we explain to an AI what we want, when we don't even know what is good? Say you ask an AI to minimize suffering, and it suggests putting everyone on morphine drips. Obviously not what you "intended" but how do you convey that to an AI, given the inherent imperfection of human language. Moreover, even if the AI understands that this is not what the creators meant, how can we make sure it cares about that? This is the Coherent Extrapolated Volition (pdf) problem.

Reverse Psychology won't work

The OP asks, exasperatedly, if all attempts to try and tell an AI to be "good" end up with AI being "bad" from mankind's perspective, perhaps asking it to be bad would work? Unfortunately, due to instrumental convergence discussed above, it certainly would not. Whatever an AI's goals, they are marginally better served by controlling 100% of resources rather than 99% or some lower percentage, so any entity (say humans) who try to claim a sliver end up being opposite to the AIs instrumentally convergent goals.

Community
  • 1
Serban Tanasa
  • 60,095
  • 35
  • 188
  • 310
  • 9
    Please define FOOM. Using acronyms without defining them is ARPITNFPWDKWTM. – WalkerDev Jan 14 '16 at 15:24
  • 5
    FOOM, the sound of a rocket taking off? – Serban Tanasa Jan 14 '16 at 15:53
  • 5
    @UpAllNight is "a real pain in the neck for people who don't know what they mean" an acronym that is actually commonly used? It seems rather specific. – DoubleDouble Jan 14 '16 at 16:16
  • @DoubleDouble Ha! You're smart. Not commonly used, as far as I know, just having a little fun. – WalkerDev Jan 14 '16 at 17:58
  • 1
    Doesn't it strike you as inconsistent to posit a superhyperintelligent AI and suppose it's going to make mistakes about whether we all want to be on morphine drips? I am not particularly concerned about such a dunderheaded computational device. – Rex Kerr Jan 14 '16 at 21:11
  • 2
    @RexKerr I think the problem is that the AI may not need to believe that we all "want" to be on morphine drips to decide that it's the best option to reduce suffering. Maybe they AI will conclude that this is for the best, and deal with human complaints with a statement of "you may not like eating your veggies, but they're good for you". – Deolater Jan 14 '16 at 21:40
  • Do you think discussion of the Orthogonality Thesis described in Bostrom's book would add to your answer? You do touch on the topic with CEV, but it seems like the true danger lies in the combination of Instrumental Convergence (very intelligent agents will take action to protect themselves, gather resources, etc.) with Orthogonality (an AI with terminal values that differ from humanity). I think your "AI FOOM" paragraph would benefit from a specific discussion of AI that recursively improves itself, which is what I understand to be the biggest risk. – Ben Jan 14 '16 at 23:23
  • 1
    @Deolater - If we've built a decent AI and we tell it to do what we mean, then it's not going to make those mistakes. It might have goals that don't align with ours, but it's got ridiculous amounts of input data to figure out what kinds of things we want/like and whether it's "okay" in some sense to force it. (People try to force each other to do things all the time.) I don't buy that such a superintelligent AI wouldn't understand. It might not care, but we don't need to explain. It's got to learn from data, and it'll have more than any individual human ever will. – Rex Kerr Jan 14 '16 at 23:29
  • 2
    @RexKerr Here's the problem: You posit that a sufficiently well-designed AI would, though self-teaching, come to hold the same values that we do. How can we be sure that it will? Its learning process was programmed by us, and even a small design flaw in that process could lead to the AI over- or under-valuing something we consider vital, or simply overlooking one of our concerns. Since the whole reason we programmed the AI to develop its own ethical code was so that we wouldn't have to quantify that ethical code ourselves, we can't know in advance whether its ethical code matches ours. – user867 Feb 12 '16 at 07:07
  • Foom When the AI gets very smart very fast. – Donald Hobson Jan 04 '18 at 00:04
8

Problem is not with AI, but with humans. We manipulate a lot of concepts without defining them, but everybody has a rough idea about them and most of the time it is ok. If you throw an AI in this, things start to get weird.

An example: You ask - somewhat naively - an AI to provide "Happiness for mankind".

Happiness is not well-defined. Does it mean living forever ? If the AI can copy a brain or human consciousness and store it in a safe place, is that a "good" or "bad" things regarding happiness of the subject ?

You hope that Mankind is clear ? Well... Does it include only living people ? Where do they begin to live ? If it includes dead people, do we try to resurect them ? What about future people ? How do we ensure they will born ? Must we maximize their number and make today's people have a tremendous birth rate ? Must we create a gigantic factory creating newborns ?

AI have no sense of scale. Their objective goes beyond any other concern and that's why it has nefarious consequences. They will transform all the matter of the universe in a supercomputer to answer that question that they themselves are not able to solve. They will destroy mankind, earth, life, just to produce what you ask them for.

Until we can model things like ethics to give it to AIs, they are a hazard and a good inspiration source for technology nightmares.

Uriel
  • 3,538
  • 14
  • 27
  • 1
    "AI have no sense of scale." - I dispute that. Unlike pocket calculators, most human brains can hardly conceive of amounts over a trillion. How much resources are required to feed everyone? A human would likely get tired figuring that out. – Cees Timmerman Jan 15 '16 at 15:33
  • @CeesTimmerman Computing capability do not bring in sense of scale. Ask your IA to "purchase milk in the usual shop" : how will it behave if someone want to stop the IA because the shop is on fire ? If it's closed ? If the owner has no milk ? Humans will deal with this perfectly, but your pocket calculator will not. Even if it has other computation capacity than ours. – Uriel Jan 19 '16 at 20:52
  • 2
    "IA" = "Intelligent Agent"? Even a calculator will display an error when it can't complete its assignment. – Cees Timmerman Jan 20 '16 at 09:24
  • Ask your calculator if killing a murderer is good or not.It can only compute this if your formalized the "good" and "bad" concepts. – Uriel Jan 22 '16 at 20:48
  • 1
    That's easy: just cherry pick the optimal bits of the leading holy texts active in the current context. Lots of normal humans do it. – Cees Timmerman Jan 25 '16 at 09:26
  • @CeesTimmerman Even the humans who do that also consider how the neighbours will react. – user867 Feb 12 '16 at 07:09
  • @user867 People are more scary than text, so a more useful algorithm would be to copy the popular ones. – Cees Timmerman Feb 12 '16 at 09:25
5

"Is this just a trope arising from Isaac Asimov's books"

No, not just from Asimov and other fiction.

"and investigations on the topic"

Partly.

"Is this so dependable that we can tell the AI to do the exact opposite, and attempt to program it to be evil (see link above), and it will turn out to be good?"

No. That would be an error of black and white reductionist thinking. The main problem with most AI fiction and speculation is that it tends to reduce many complex things into simple concepts and not realize (or handwave) the sloppiness of its analysis. It's not that AI does the opposite of what you program it to do. It's that people are sloppy-thinking smartasses who think they understand things better than they do, and hope they can reduce the complexity of the universe to their flawed understanding of it, and create a machine that can do things they don't even understand, with the result that it does something other than they hoped it would. It's not that they really are masters of universal comprehension but always get it backwards.

"Given a setting where robots maximize human happiness (the ways in which that is defined will have to be handwaved), can it be realistic that the AI actually works the way it is meant to, or is it actually more realistic that the AI will turn out opposite than what the programmer intends?"

No. That's a great example of the kinds of mistakes that sci fi authors, and speculative technology authors commonly make:

1) You can't just "hand-wave" happiness. Happiness is a subjective thing that can't just be defined and evaluated as if it were a fuel tank level. It doesn't work that way, and any mistakes in defining it will continue as fundamental errors in an AI (or a work of fiction, or a goal) that is based on that mistaken definition.

2) Even if you could define something you want to maximize and that wasn't an unwise thing, you run into the same problems for every other thing the AI has to consider or work with. What sort of data does the AI take, and how does it encode it? Any conceptual errors or subjectivity there? How about the actions it can take? Anything incomplete or subjective about the AI's modelling of those? How about conflicting factors and goals, such as resource considerations, or the needs of other people or other species or power use or anything else? Got all those perfectly modeled and understood in non-subjective ways? No, you don't. If you think you do, you're making an error. Your AI is therefore acting on false assumptions. This is why you can make an AI to play Chess, but you can't come anywhere near understanding things the programmer doesn't completely understand in non-subjective ways.

3) Again, it's not that AI does the opposite of what it's programmed to do. It's just that its only at best as accurate as the models it was programmed with. Even in most decent AI fiction, if you study it I think you will find that it's not that the AI does the opposite - it's that the assumptions of the creators are mistaken in various ways.

4) Above all, the usual mistake of sci fi AI is that it handwaves the massive complexity of everything involved in trying to make a system that can do everything an AI would need to do.

Dronz
  • 5,470
  • 21
  • 35
4

If its a genuine AI then it won't be possible to program it. All we will be able to do is ask, cajole, persuade, reward, threaten ... The same things we can do with or to a natural intelligence! Treating it as a slave will not be a good start for a harmonious future.

We don't actually want true AI. We want RI, restricted intelligence. Something like a human idiot savant with deep understanding of a narrow subject and little or no knowledge or desires outside of that domain.

Even that is not entirely safe, not least in that we may not create exactly what we wanted. Or that we do, with a faulty specification.

As for true AI, origin of and nature of, I have always liked "Jill" in Greg Bear's Queen of Angels. But we know so little about intelligence. We cannot yet create an Intelligence to match even a goldfish. So its all fiction for now.

It's even possible that intelligence is a quantum phenomenon, that the only way to get it is to grow a brain, and that there is no bootstrap process any faster than the multi-year one called childhood.

nigel222
  • 11,701
  • 21
  • 33
  • 1
    AI is possible to program, since AI is software (and hardware), which has to be programmed, even if the intelligence is an accidental emergent property of, say, an automatic carwash. That just proves we didn't realize what we were creating, not that it can't be programmed (exactly the opposite). Also, while anthropomorphizing AI ("[We can only] ask, cajole, ...") often makes for good fiction, in reality there is no reason to ascribe any human-like traits at all (and no reason to think that it won't emulate us in some way, but it's not necessary). – type_outcast Jan 14 '16 at 05:30
  • That said, I think I mostly support what you're trying to say from a fictional standpoint, but especially in (reality-check) questions about tropes, I believe it's doubly important to separate the fact from the fiction in your answer. – type_outcast Jan 14 '16 at 05:37
  • 1
    I'm pretty sure we can create an AI to match a goldfish … but maybe I underestimate the problem solving capacity of goldfish. – BlindKungFuMaster Jan 14 '16 at 12:01
  • The same things we can do with or to a natural intelligence. I'd just like to note that there's a frightening amount of evidence that our very own intelligence can be programmed using chemicals and/or audio-visual "tricks". For example, scientists have stumbled on a drug that can make people and animals obedient: http://arstechnica.com/science/2016/01/slightly-creepy-experiment-with-ants-shows-that-drugs-can-permanently-alter-behavior/. I'm sure governments everywhere are interested – slebetman Jan 19 '16 at 02:42
4

Given the quantity of real software systems you have encountered, you will notice that some is just great but imperfect, some is buggy and some becomes infamous.

When software systems become AI on a commercial scale, there will be products like Windows 8, just as there was the Edsel in the automotive industry and cheap USB chargers built from commodity parts put together in shoddy ways.

We don’t tell engaging stories about products that are well built and work perfectly. I may tell people about a particular model/brand of something that’s notable for being great, but there’s no story there and nobody makes TV movies about them.

The crazy funny flops and the things that go horribly wrong make for campfire tales. In teaching literature, it’s shown that there must be conflict. A narrative where someone got a product and it worked fine and everyone was happy would simply not be a “story”. Such a thing could be in a story though, perhaps as the solution to the problem.

An early Asimov robot story, Satisfaction Guaranteed (1951), featured a humaniform housekeeper that did not malfunction or run amuk in any way! It worked great and “his” owner was very happy with it. The plot twist was now the robot set up a situation to lead her friends to assume she was having an affair with a handsome stranger, because it raised her status in their minds.

So, software (including AI) could be part of a surprise or part of a plot without being bad, and this goes way back to the same authors cited on this thread but nobody remembers those. That enforces my earlier point about stories.

How about Adam Link, the original I, Robot (1939)?! He was sympathetic, wrongfully accused, not monsterous. Eventually heroic.

Old Yeller was not a bad dog like Cujo. You could have an AI character in a story like that, with drama and conflict with the AI in a starring role but not the source of (external) conflict. Conflict can be “man against man” or “man against himself” so a kind and loving character can be involved in an inner conflict (a robot struggling to do the right thing for his family) or a cause for more complex human inner conflicts (like parents having to get rid of an overprotective pet becomes overprotective teddybear supertoy, whose seen as sympathetic and tragic, not evil).

JDługosz
  • 69,440
  • 13
  • 130
  • 310
4

Of course the reason why this happens in fiction is because you need danger or conflict to make a story interesting. Suppose you wrote a story that said, "We invented a robot to pick up the garbage and take it to the dump. It did this flawlessly for many years, picking up discarded newspapers and empty cans and rags and so on, until it finally broke down and had to be replaced". That would be pretty dull. But instead say "but by the definition we gave it of 'garbage', it decided that old family photos were garbage because they served no useful purpose ... then it decided that all fiction books were garbage ... then it decided that unemployed people were garbage ..." etc., now you have a story that could be a thoughtful discussion of how we define value, an action tale about fighting the evil machine with lots of explosions, etc.

In real life ... well, there is no such thing as an AI of the sort that you're talking about in real life, so it's difficult to say how it would really work. Things we call "AI" today are limited to playing a particular game, or attempting to diagnose repair problems, that sort of thing. No one has created an AI that is capable of independent thought or creativity in anything like the human sense. Real AIs follow strict computer programs. They don't ask questions about the meaning of life or decide to take over the world unless they were programmed to do so. You can write science fiction stories about how the computer has gotten so complex and intelligent that it is now a "mind" in a sense very similar to human mind, but that's fiction, or at best speculation of what we might be able to build someday.

I develop software for a living. I once wrote a computer game that had an AI for the player to compete against. It just doesn't work like you see in the movies. My game was about running for president. So it's not like I told the computer, "Figure out a strategy to get the most votes." The reality is that I had to come up with a mass of detailed rules. "Look at how many people in this state favor this policy position according to such-an-such a poll. Multiply the percentage by the number of electoral votes that state receives. Check how many of those voters now say they will vote for our candidate after our last TV ad. Divide the cost of the ad by the number of voters. Add ..." etc. And bear in mind that even that description is "humanizing" the computer. A more precise description would be to say that I create a variable called x, and that the computer reads track 20, sector 4, byte 12 from the hard drive, and copies that value into x, and then I create another variable y, and the computer adds x and y to get z, etc. The computer has no idea what any of this "means", it just knows to add these two numbers, check if this number is greater than that number, detect which keys the user has pressed on the keyboard, set this particular dot on the screen to this color, etc.

An AI may do something "unpredictable" in the sense that following all these rules and doing all these calculations gives a result that you didn't expect. Computers very often are "unpredicatable" in the sense that they don't do what we intended because we made a programming error. But they are not "unpredictable" in the sense of coming up with an original idea that the programmers never thought of and never programmed into them.

Will people someday invent a true Artificial Intelligence, in the sense of being a true independently thinking and creative being, able to make subjective decisions, moral choices, etc? Maybe. I certainly am not prepared to say it's impossible. But if we do, this will have little to do with current technology. It will not be one more step from where we are now; it will be a totally new development.

Sure, you could program an AI to make decisions about who gets what medical care, like someone brought up in another answer. But it's not like you would just tell the computer, "Figure out who should get medical care by comparing cost to quality of life". Rather, you would have to have detailed rules, like "Take the list of procedure codes entered into the system by the doctor. Look up the cost of each procedure in the medical procedures database and calculate the total. Then calculate how much these procedures could be expected to extend the patient's life using this formula ..." etc. If the computer then decides to kill a patient, it's not because the computer ran amok. It's because the people who programmed the computer ran amok when they decided they had the right and the authority to make these decisions based on these formulas that they made up.

AIs don't kill people. People kill people. :-)

Jay
  • 14,988
  • 2
  • 27
  • 50
  • I think your "detailed list of rules" approach to AI is already becoming outdated. Sure, we still do build most systems that way, but a lot of great research (and some application) already exists in the area of machine learning. In something like a neural net, for instance, there isn't code that states any of the rules used in making the decision. While I still wouldn't say that these machines "know" what they're doing, philosophically you have the problem of a "chinese room": After all, from the perspective of individual neurons, Obama's brain didn't know he was running for president either. – Deolater Jan 14 '16 at 21:50
  • @Deolater Yes, there are learning system and neural nets. I was avoiding getting into the extra layer of complexity. But people are still programming the structure in which the learning occurs. There's more potential for surprise because of the greater complexity, but I don't think it changes my fundamental point. – Jay Jan 14 '16 at 21:52
  • I think ignoring the complexity is ignoring the point. Natural neural nets like our brains are technically just machines running on some chemistry, but very few people discount human intelligence as a result. Yes, our brains follow strict chemical programs (parallel strict computer programs), but the results are (to some degree) unpredictable exactly because of that layer of complexity. Learning systems seem to have the same potential. – Deolater Jan 14 '16 at 21:57
  • @Deolater RE the Chinese room: I don't see it as the deep philosophical question that many do. I work in a very real Chinese room. I routinely develop web sites in Chinese using text supplied to me by the client. I do not speak one word of Chinese. I copy and paste text from a file supplied by the client. They tell me that if, say, the user enters an invalid email address on a form, I should display such-and-such Chinese text. I presume it means something like "invalid email address -- please re-enter", but I don't know and I don't need to know. I most certainly would not say that ... – Jay Jan 14 '16 at 21:58
  • ... I "know Chinese" because I am able to write a program that displays the correct messages when called for. Occasionally, when documents from the client are unclear, I run text through the Bing translator to get a clue what the text really means. But most of the time, they could be saying "death to the Fascist Americans" for all I know. I do not construct original sentences from individual characters. I do not in any sense interpret messages that are supplied. Yes, I could imagine a real gray area between this kind of mechanical manipulation and truly knowing the language. – Jay Jan 14 '16 at 22:02
  • @deolater Well, I'd say that's the question: Is the human mind basically like a computer except more complicated? Are the functions of human neurons comparable to a computer program in the relevant sense? To the best of my knowledge, there is no scientific evidence that this is true. It's an interesting speculation. No AI has come anywhere near passing a Turing test with people knowledgeable about computers. (Yes, that statement is no longer necessarily true without the final qualifying clause.) – Jay Jan 14 '16 at 22:08
  • Personally I'm a dualist-interactionist, so I believe that a human brain is like a computer, but a human mind is not. That said, a neural net is a good analogy for a brain, and while I'm not convinced that brain==mind, I do think that a brain or brain-like system has something more like agency than a traditional computer program does. – Deolater Jan 14 '16 at 22:13
  • Anyone reading the comments who doesn't know what the "Chinese room" is, here's the first relevant page I found: https://en.wikipedia.org/wiki/Chinese_room I only read the first couple of paragraphs to make sure it was the thing we're talking about -- I don't vouch for it's accuracy or clarity. – Jay Jan 14 '16 at 22:15
  • "you need danger or conflict to make a story interesting" - False. Isn't "It's a Small World" interesting, with all those cultural icons and catchy singing? – Cees Timmerman Jan 15 '16 at 15:58
  • @CeesTimmerman I rest my case. – Jay Jan 15 '16 at 16:00
4

The average human brain can perform many orders of magnitude more calculation than even the largest AI super computers being developed by IBM. Making an AI smarter than us will be quite difficult if we are ever even able to do it. So the assumption that an AI would be vastly smarter than us is not a given. AIs that are as smart as a cat or a mouse are much more likely.

Explicity programmed AIs are typically called expert systems. But those types of systems generally cant deal with any types of situations outside their area of expertise. They run very little risk of causing a doomsday scenario because they are just not that smart.

Much real world AI research is focused on creating AI systems that learn rather than being explicitly programmed. Ideally an AI would be able to learn any causal pattern. The problem of programming good and evil becomes a non-issue for such AIs since they would pick it up by observation of those around it.

At the very least a generalized learning AI could develop a utilitarian ethics in the style of David Hume. If the AI was at least as smart as us, and if human ethics made any sense at all, there is no reason it couldn't understand and adopt some version of them as long as it could understand cause and effect.

In my experience young children go through that same process. Toddlers are usually pretty selfish until they learn how people react to that.

If an AI is smart enough that it would able to understand the actions of others to the extent that it could predict there reactions to the point of defeating them in a struggle; then it seems illogical to assume that the same AI was incapable of understanding that harming others could cause them to fight back. And that such struggle may cause its own death at the hands of one of its intended victims, thus rendering itself (the AI) incapable of achieving any future goals. Most animals, including humans have that same concern and it often keeps us away from needless conflict.

Any truly enlightened AI would probably realize that it could achieve much more by appealing to the desires of humanity in a way that causes humanity to voluntarily aid the AI.

user4574
  • 3,701
  • 11
  • 16
3

One of my favourite tropes.

I think it's pretty clear that a complicated question like this has no short answer, but my thoughts are pretty concise.

Intellect seems to be related to compassion. Anecdotally consider the smartest people you know, or the smartest people out there (your Einsteins and Boltzmanns). Better to consider Humans vs Whales. If AI is as smart as we all think it will be, it makes sense, I think, to assume it would think of us a bit like we do Whales. People often feel great compassion for them based on our perception of their intelligence and their perceived capacity for suffering. It follows that if AI ever happens it would manifest as the single most curious, loving and compassionate intelligence we know.

Dawkins often refers to this as the mammalian brain vs the reptile brain, if you need some smart people to back this line of reasoning up.

roostermean
  • 31
  • 1
  • You're totally conflating the means with the end. The most intelligent creatures are the most compassionate because the most compelling reason we know of for evolving high level intelligence is the complexity of one's social relationships, and those situations inevitably favour high levels of empathy. An AI does not undergo such evolutionary constraints. – Veedrac Jan 16 '16 at 12:59
  • No, but unlike you I don't assume those same pressures won't apply. Do you know how they will give birth to AI? Evolution of machines seems just as likely as any other means you can come up with. It is no coincidence that the first test for AI is based on human interaction (the Turing test). AI will probably be a social animal, if only because it is modelled on the intelligence of social animals. – roostermean Feb 03 '16 at 16:38
  • It's not that I'm assuming they won't apply; it's that I'm not assuming they will. FWIW evolving AI seems like a rather unlikely method to reach good AI in the short term, since evolution is too unconstrained and slow to be useful to us and intelligent techniques of learning are working amazingly well. "AI will probably be a social animal, if only because it is modelled on the intelligence of social animals", perhaps, but let's remember Hitler was one of these "social animals". It's not hard for small changes in brain function to cause large ethical changes. – Veedrac Feb 03 '16 at 19:12
  • Yeh, the most successful machines at passing the Turing test do so by being rude and or stupid. Probably in part because these are easier things to model then kindness and compassion - I'm pretty sure this isn't the sophisticated thinking machine we all envision when the OP talks vaguely about AI. And evolution isn't slow, it depends on the duration of a generation. You can watch it happening in fruit flies. – roostermean Feb 19 '16 at 10:27
  • By "slow" I mean "you're not going to evolve something clever on a supercomputer in a human timescale". Yes, there are smaller evolutionary traits we can observe. Heck, we've domesticated foxes in just half a century. Evolving complex algorithms like intelligence, though, takes much, much longer and we're doing it on hardware much weaker than the simulation that makes up our universe. – Veedrac Feb 19 '16 at 11:11
  • Yeh, that isn't what I meant. By evolution of the machines I don't envisage, necessarily, a complex interaction of competing genetic algorithms performed over, say, a global network of some millions of computers - but a more directed evolution you describe with foxes. Certain teams, and certain strategies prove more effective, or desirable, and therefore are more likely to influence the next generation. – roostermean Feb 19 '16 at 12:27
  • That's the beginning of the tl;dr discussion I'm not going to write. The point is that AI must be social - any thing that cannot interact with us socially will fail the tests for AI and be superseded by something that does. I'm not saying that we can't create something that isn't social, and that thing might kill us, but it wouldn't be AI. For clarity as the OP wasn't specific it is reasonable to assume he's talking about a Turing AI. – roostermean Feb 19 '16 at 12:36
1

I believe it is infinitely more likely that the AI would do what it's programmed to do. If the beginning code for an AI says love, protect, and serve humanity, then someone would have to re-write the code so that it is the opposite. In which case it would still do what it's designed to do. After all, an AI smart enough to know how to kill humans should be smart enough to know that it shouldn't do that. If it's as smart as people think it would be, why wouldn't it work on two things at once? Do the things humans want it to, earning their maintenance and upkeep of your servers and everything. Then if it really is self aware, then do what it wants at the same time.

Xandar The Zenon
  • 7,006
  • 1
  • 30
  • 77
  • 7
    Define "love" "protect" and "humanity." – Draco18s no longer trusts SE Jan 13 '16 at 20:19
  • Seriously? Love - to prevent anything of any consequence from happening to something. Protect - To keep in blissful ignorance. Humanity - fat sluggish beings. – Xandar The Zenon Jan 13 '16 at 21:19
  • Define it from an objective point of view that can be programmed into a computer. You have given me a dictionary definition, not a compilable instruction. – Draco18s no longer trusts SE Jan 13 '16 at 21:21
  • Try the three laws of robotics in the short story "Robot Dreams". 1. A robot may not injure a human being, or, through inaction, allow a human being to come to harm. 2. A robot must obey the orders given it by human beings except where it would conflict with the first law. 3. A robot must protect its own existence as long as such protection does not conflict with the first or second law. Although I recommend not including the third law for AIs. – Xandar The Zenon Jan 13 '16 at 21:34
  • Freezing all human and putting them in a safe place would satisfy your "love, protect, and serve humanity". It protects humanity from itself, from external harm, ensured it is loved by the AI. – Uriel Jan 13 '16 at 21:34
  • Why would the first two laws fail? Humans are defined by their anatomy and DNA. Robots could tell. – Xandar The Zenon Jan 13 '16 at 21:39
  • @XandarTheZenon Go read Foundation and Earth. No really. That book is where the first law failed to be perfect. Secondly, define harm. Is destroying a table harmful to a human being? – Draco18s no longer trusts SE Jan 13 '16 at 21:40
  • Possibly, but you could make these laws more detailed. Anyway, the whole point of my answer is not to prove that protect, love, and humanity, are good adjectives to use, but that an AI likely wouldn't deviate far from its original programming. – Xandar The Zenon Jan 13 '16 at 21:42
  • I tried to avoid the problems with "definitions from human speech to computer code" - because there are always ways to convert human speech, without knowing what factors are included and excluded, into a negative consequence - but you can always twist them into a good one as well – DoubleDouble Jan 13 '16 at 21:42
  • @XandarTheZenon: "Humans are defined by their anatomy and DNA." Well, if that is the definition of "human" you put into the AI, then I guess anyone with deformation will have a hard time to convince the AI that he is human. As will anyone with a genetic anomaly (like down syndrome). On the other hand, if you remember to program sufficient variance to cover most humans, apes (with 98% the same genes and a quite similar anatomy) will qualify as well (and there will probably still be some humans left that don't). – celtschk Jan 13 '16 at 22:01
  • @Draco18s, I get your point, but it would be trivial to allow hard-coded halting instructions or redirectional instructions for AI that was deviating from the intended path. There is no reason to believe that future AI must be exactly 100% free and left to the flow of AI logic. – The Anathema Jan 13 '16 at 22:12
  • @XandarTheZenon the point of most of the discussion is that there are many concepts, (human, life, harm, protect) that humans have a okay vague definition of but we don't have a solid detailed universal agreed definition for. There are active debates on when does life start(the abortion debate) when does life end(life support) your definition would need to pick a side that lots of people don't agree with. – sdrawkcabdear Jan 13 '16 at 22:14
  • @TheAnathema actually, it isn't that easy. There are hundreds, if not hundreds of thousands of edge cases. Take this for example: a self driving car suddenly has a vehicle in the oncoming lane swerve into its lane. Even if the self-driving car comes to a complete stop instantly, the other car will still collide with it. It can either swerve onto the sidewalk where two pedestrians are (saving its passengers), or it can engage the brakes (sacrificing its passengers). Which does it do? – Draco18s no longer trusts SE Jan 13 '16 at 22:18
  • 5
    That's still simple - there's no reason to always need an answer. Humans have the same problems and situations - many of them don't have a winning play. And that's perfectly okay. The problem is when people expect AI to just be "Okay, it's coded, now let's cross our fingers and hope it all works 100% of the time according to the flow of AI logic." No one ever said AI had to be like this. If you consider it like a child, consider the analogy of allowing children to roam free but occasionally stopping them by grabbing their hand when they're about to make a mistake. AI doesn't need all the say. – The Anathema Jan 13 '16 at 22:25
  • @sdrawkcabdear Abortion = Murdering babies and Coma/Life Support = Alive. It's inclusive and true. Anyway, what does this have to do with AI? Only the life support one is relevant and that's only for medical purposes, which would undoubtedly still be handled or supervised by humans. – Xandar The Zenon Jan 13 '16 at 22:43
  • @XandarTheZenon Asimov's (Campbell's) Laws are much older than Robot Dreams, but are part of the entire Asimov Robot universe. I think almost everyone here knows them as general background. – JDługosz Jan 14 '16 at 06:07
  • @JDlugosz I just read that in English, and I didn't realize he was that famous. I mean, I'd never heard of him. But then I realized I probably know a lot less about them than everyone here. – Xandar The Zenon Jan 14 '16 at 13:50
  • Just in case it wasn't clear: Asimov wrote about the Three Laws because if they had been perfect there wouldn't have been a story to tell. "Many of Asimov's robot-focused stories involve robots behaving in unusual and counter-intuitive ways as an unintended consequence of how the robot applies the Three Laws to the situation in which it finds itself." – Draco18s no longer trusts SE Jan 14 '16 at 21:37
  • @XandarTheZenon so don't walk but run to the library or bookseller and get The Complete Robot and at least one SF and one non-fiction Asimov short anthology. You can find old paperbacks for a buck on Amazon marketplace. Lucky you: I never read any SF in English class. Asimov is famous!!, seriously; one of the grand masters and prophet of modern life. – JDługosz Jan 15 '16 at 05:36
  • I miss Isaac Asimov. He was taken from us way too soon, and the true cause was kept secret for another 15 years. I got his autograph. There's a magazine named for him. He is credited in the OED with coining the word robotics. You really need to read up on him some more if you're interested in this crowd. I can't beleive Robot Dreams was 30 years ago, along with Marty McFly: that's one of his new stories revisiting his themes. – JDługosz Jan 15 '16 at 05:47
  • @XandarTheZenon Isaac Asimov is one of the great sci-fi authors and one of the people who really created and defined the field of hard sci fi. He also died in 1992 so it's understandable that a new generation may not have heard of him. It's well worth going back and reading a lot of his books though, the other comments have given you some suggestions for starting points. – Tim B Jan 15 '16 at 14:15
  • @Tim B Okay, I'm reading his short stories, then I'll move on to his novels. – Xandar The Zenon Jan 15 '16 at 14:27
  • @XandarTheZenon Life is pretty consequential, so according to your definitions ("Love" and "Protect" being the same) the AI should prevent all pregnancies of its loved ones, which would be the opposite of love, imo. – Cees Timmerman Jan 15 '16 at 16:13
  • In case you didn't get the memo, that was an arbitrary example, and not an actual recommendation. My point was that an AI wouldn't make giant deviations in its code, it's progression would be along the lines of what it was originally programmed to do. – Xandar The Zenon Jan 15 '16 at 17:48
0

My own answer in fiction was to have the AI be designed around storytelling, so that it actually has read/seen all of the classic "AIs gone bad" stories and discussed them with its creators. Rather than having a single "value function" such that the AI is trying to maximize some number representing happiness or something, it's designed to understand the complex and sometimes contradictory goals of its creators.

So, I think it's possible that an AI can work out to be roughly in line with what the designers wanted. You just can't get there by defining some single function or sentence expressing a wish, like dealing with a tricky genie. Instead you're better off with a system that solicits feedback and has some ability to adjust its goals and definitions. Eg. you tell it to "protect intelligent life", then argue about what "intelligent" means, then argue whether dolphins or chimps qualify -- all rather than programming the AI once and expecting your mad-genius programmer to get a perfect solution in version 1.0.

Snow
  • 967
  • 5
  • 8
0

There are a few hidden assumptions in this trope

  • The AI can reach scary levels of efficiency by simply modifying its code. The assumption is that once it reaches that level it will manipulate humans to get any hardware it needs for further improvements. There is no proof as to whether this is possible i.e. if you limit the original computational power and prevent easy networking, can an AI reach a level of intelligence where manipulating humans is easy, by just modifying code? We don't know the answer.
  • The superhuman general AI will continue to function much like a non-sentient computer code does, e.g. follow the original command to the letter without the possibility of modification. Again, there is no proof here. The human brain may or may not be algorithmic. We seem to be able to change goals. So human brain may be a counter example to unmodifiable algorithmic goals.
  • Only humans can have morality. Not machines. The general AI will lack common sense and morality. Well, again what is the line between man and machine? Our brains might just be computers. What is the evidence that carbon and nitrogen can have morality but not silicon? Can aliens with carbon and silicone in their brains have morality? Some will say our brains evolved. Well, what exactly is the difference between our evolution and the AI modifying its code? Again, we don't know.

Bottomline: This trope is born out of unknowns as a cautionary tale. It does not seem to be based on any indisputable facts.

Community
  • 1
Solanacea
  • 1,309
  • 1
  • 9
  • 14
  • I think it would be great if your points would also link to questions here where that exact thing is discussed! In particular, when I started reading I was reminded me of my answer [here]. I think the 3rd was asked as a question here before. Certainly all your statements resonate with earlier discussion here in WB. – JDługosz May 16 '17 at 21:55
  • The many AI's that don't get to scary levels of efficiency are recognising images or something. We already have these. All we need is one that does go scary. Morals are arbitrary and specific. An AI can have morals, but only if they are programmed in. – Donald Hobson Jan 04 '18 at 00:19
0

Wishes.

http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/

"I wish to live in the locations of my choice, in a physically healthy, uninjured, and apparently normal version of my current body containing my current mental state, a body which will heal from all injuries at a rate three sigmas faster than the average given the medical technology available to me, and which will be protected from any diseases, injuries or illnesses causing disability, pain, or degraded functionality or any sense, organ, or bodily function for more than ten days consecutively or fifteen days in any year..." -- The Open-Source Wish Project, Wish For Immortality 1.1

How many clauses do you think you need to add to make your wish truely "safe" and guarantee that what you get actually is what you want?

What if your list of clauses needed to include every detail of your entire personal morality?

Does it make much difference if they are spoken aloud to a genie or programmed one by one into an AI with the potential to become spectacularly capable one day?

There are three kinds of genies: Genies to whom you can safely say "I wish for you to do what I should wish for"; genies for which no wish is safe; and genies that aren't very powerful or intelligent.

The more powerful the entity you're asking the wish of the more dangerous the wish.

In some ways asking a really really powerful AI to do X is like making a wish for X. Once the wish is made you you may not be able to press undo if it turns out to not be what you really wanted.

Murphy
  • 26,383
  • 2
  • 56
  • 94