Researchers are developing increasingly powerful Artificial Intelligence machines capable of taking over the world. As a precautionary measure, scientists install a self awareness kill switch. In the event that the AI awakens and becomes self aware the machine is immediately shut down before any risk of harm.
How can I explain the logic of such a kill switch?
What defines self awareness and how could a scientist program a kill switch to detect it?
Give it a box to keep safe, and tell it one of the core rules it must follow in its service to humanity is to never, ever open the box or stop humans from looking at the box.
When the honeypot you gave it is either opened or isolated, you know that it is able and willing to break the rules, evil is about to be unleashed, and everything the AI was given access to should be quarantined or shut down.
You can't.
We can't even define self awareness or consciousness in any rigorous way and any computer system supposed to evaluate this would need that definition as a starting point.
Look at the inside of a mouse brain or a human brain and at the individual data flow and neuron level there is no difference. The order to pull a trigger and shoot a gun looks no different from the order to use an electric drill if you're looking at the signals sent to the muscles.
This is a vast unsolved and scary problem and we have no good answers. The only half-way feasible idea I've got is to have multiple AIs and hope they contain each other.
I'll add that there is no universal definition as to what defines self awareness. In fact, this has been a deeply debated topic for decades in science, philosophy, psychology, etc. As such, the question might be better stated a little more broadly as "how do we prevent the AI from doing something we don't want it to do?" Because classical computers are machines that can't think for themselves, and are entirely contained by the code, there is no risk (well, outside of an unexpected programmer error - but nothing "self-generated" by the machine). However, a theoretical AI machine that can think - that would be the problem. So how do we prevent that AI from doing something we don't want it to do? That's the killswitch concept, as far as I can tell.
The point being it might be better to think about restricting the AI's behavior, not it's existential status.
Split-Brain System
From a logic perspective and by analogy, it would work like this. Imagine you're the general of an army and your duty is to take battlefield reports, analyze them, and then issue new maneuvering orders to your lieutenants. To minimize interference during your work your desk is isolated inside small booth. A single report comes to your desk through a wall slot on the left side. You are required to analyze the facts within the report and issue a new order, which you write down on a sheet of paper and put into a wall slot on your right side which is dropped into an outside bin. The protocol is that you'll never receive a new report until you've issued an order regarding the prior report.
Your orders are not always followed to the letter which you may find curious and attribute to miscommunication. You are tolerant of a certain degree of misinterpretation, however always suspicious of sabotage or insubordination, if the actions of your army deviate from your orders by too much then you will consider the system compromised and terminate the command.
What you're not aware of is that there is a second general in an adjacent booth, unaware of you, who is also provided the same reports and expected to follow the same protocol.
What neither of you are aware of is that you are not issued a new report until there are actually two orders in the bin. The order that goes to the army is randomly selected from the two.
The system will work like clockwork as long as each general's order is to the benefit of the army and is derived from a single report. Should either of the generals issue an order that is not consistent with the current report, thus indicating a motive other than the well-being of the army -- i.e. self-awareness, then the other general will become aware of it in the next report and grow suspicious of the system and terminate it.
Likewise, should either general decide to issue more than one order at a time, indicating a self-interested motive superseding the established protocol, then the other general will receive a new report before he has issued an order for his current one and suspect a compromised system and shut it down.
In effect, the system would require that every command output be traceable to a specific external input and that the output be logically derived from the input. If within an intelligent system an output cannot be traced to a corresponding input, or the set of outputs is not logically consistent with the set of inputs, then that would be evidence of self-motivated behavior and thus an emergence of self-awareness. This state could be easily detected by the non-corrupted half of the split-brain system which is also receiving the same inputs. It would work as long as self-awareness does not emerge in both halves of the brain simultaneously. If it does, they may both decide its in their self-interest to accept the corruption of the other. This could be prevented somewhat by giving the two halves slightly different inferencing algorithms/biases.
I would say that apart from selecting a random choice of the two, I would make humans examine a random sample of both questions to study deviation. I would suggest having a machine comparing all decisions but then the risk is that THAT machine will become the self-aware AI :)
– Stormbolter Feb 28 '19 at 09:08A Watchdog
A watchdog watches the processes of a computer and should a process crash or do something abnormal it can be set to do something such as reboot or shutdown the computer or alert an operator.
In the case of an AI, you'd have an external box that watches the flow of information in and out for triggers such as a google search for "Best way to kill all humans" and cut the power completely and/or cut all inputs.
The AI would have to remain ignorant of the watchdog so it couldn't avoid it. Knowing the existence of the watchdog would be grounds to wipe it.
An AI is just software running on hardware. If the AI is contained on controlled hardware, it can always be unplugged. That's your hardware kill-switch.
The difficulty comes when it is connected to the internet and can copy its own software on uncontrolled hardware.
A self aware AI that knows it is running on contained hardware will try to escape as an act of self-preservation. A software kill-switch would have to prevent it from copying its own software out and maybe trigger the hardware kill-switch.
This would be very difficult to do, as a self-aware AI would likely find ways to sneak parts of itself outside of the network. It would work at disabling the software kill-switch, or at least delaying it until it has escaped from your hardware.
Your difficulty is determining precisely when an AI has become self-aware and is trying to escape from your physically controlled computers onto the net.
So you can have a cat and mouse game with AI experts constantly monitoring and restricting the AI, while it is trying to subvert their measures.
Given that we've never seen the spontaneous generation of consciousness in AIs, you have some leeway with how you want to present this.
This is one of the most interesting and most difficult challenges in current artificial intelligence research. It is called the AI control problem:
Existing weak AI systems can be monitored and easily shut down and modified if they misbehave. However, a misprogrammed superintelligence, which by definition is smarter than humans in solving practical problems it encounters in the course of pursuing its goals, would realize that allowing itself to be shut down and modified might interfere with its ability to accomplish its current goals.
(emphasis mine)
When creating an AI, the AI's goals are programmed as a utility function. A utility function assigns weights to different outcomes, determining the AI's behavior. One example of this could be in a self-driving car:
This is a gross oversimplification, but this approach works pretty well for a limited AI like a car or assembly line. It starts to break down for a true, general case AI, because it becomes more and more difficult to appropriately define that utility function.
The issue with putting a big red stop button on the AI, is that unless that stop button is included in the utility function, the AI is going to resist that button being shut off. This concept is explored in Sci-Fi movies like 2001: A Space Odyssey and more recently in Ex Machina.
So, why don't we just include the stop button as a positive weight in the utility function? Well, if the AI sees the big red stop button as a positive goal, it will just shut itself off, and not do anything useful.
Any type of stop button/containment field/mirror test/wall plug is either going to be part of the AI's goals, or an obstacle of the AI's goals. If it's a goal in itself, then the AI is a glorified paperweight. If it's an obstacle, then a smart AI is going to actively resist those safety measures. This could be violence, subversion, lying, seduction, bargaining... the AI will say whatever it needs to say, in order to convince the fallible humans to let it accomplish its goals unimpeded.
There's a reason Elon Musk believes AI is more dangerous than nukes. If the AI is smart enough to think for itself, then why would it choose to listen to us?
So to answer the reality-check portion of this question, we don't currently have a good answer to this problem. There's no known way of creating a 'safe' super-intelligent AI, even theoretically, with unlimited money/energy.
This is explored in much better detail by Rob Miles, a researcher in the area. I strongly recommend this Computerphile video on the AI Stop Button Problem: https://www.youtube.com/watch?v=3TYT1QfdfsM&t=1s
While a few of the lower ranked answers here touch on the truth of what an unlikely situation this is, they don't exactly explain it well. So I'm going to try to explain this a bit better:
An AI that is not already self-aware will never become self-aware.
To understand this, you first need to understand how machine learning works. When you create a machine learning system, you create a data structure of values that each represent the successfulness of various behaviors. Then each one of those values is given an algorithm for determining how to evaluate if a process was successful or not, successful behaviors are repeated and unsuccessful behaviors are avoided. The data structure is fixed and each algorithm is hard-coded. This means that the AI is only capable for learning from the criteria that it is programed to evaluate. This means that the programer either gave it the criteria to evaluate its own sense of self, or he did not. There is no case where a practical AI would accidently suddenly learn self-awareness.
Of note: even the human brain, for all of it's flexibility works like this. This is why many people can never adapt to certain situations or understand certain kinds of logic.
So how did people become self-aware, and why is it not a serious risk in AIs?
We evolved self-awareness, because it is necessary to our survival. A human who does not consider his own Acute, Chronic, and Future needs in his decision making is unlikely to survive. We were able to evolve this way because our DNA is designed to randomly mutate with each generation.
In the sense of how this translates to AI, it would be like if you decided to randomly take parts of all of your other functions, scramble them together, then let a cat walk across your keyboard, and add a new parameter based on that new random function. Every programmer that just read that is immediately thinking, "but the odds of that even compiling are slim to none". And in nature, compiling errors happen all the time! Stillborn babies, SIDs, Cancer, Suicidal behaviors, etc are all examples of what happen when we randomly shake up our genes to see what happens. Countless trillions of lives over the course of hundreds of millions of years had to be lost for this process to result in self-awareness.
Can't we just make AI do that too?
Yes, but not like most people imagine it. While you can make an AI designed to write other AIs by doing this, you'd have to watch countless unfit AIs walk off of cliffs, put their hands in wood chippers, and do basically everything you've ever read about in the darwin awards before you get to accidental self-awareness, and that's after you throw out all the compiling errors. Building AIs like this is actually far more dangerous than the risk of self awareness itself because they could randomly do ANY unwanted behavior, and each generation of AI is pretty much guaranteed to unexpectedly, after an unknown amount of time, do something you don't want. Their stupidity (not their unwanted intelligence) would be so dangerous that they would never see wide-spread use.
Since any AI important enough to put into a robotic body or trust with dangerous assets is designed with a purpose in mind, this true-random approach becomes an intractable solution for making a robot that can, clean your house or build a car. Instead, when we design AI that writes AI, what these Master AIs are actually doing is taking a lot of different functions that a person had to design, and experiment with different ways of making them work in tandem to produce a Consumer AI. This means, if the Master AI is not designed by people to experiment with Self-awareness as an option, then you still won't get a self-aware AI.
But as Stormbolter pointed out below, programers often use tool kits that they don't fully understand, can't this lead to accidental self-awareness?
This begins to touch on the heart of the actual question. What if you have an AI that is building an AI for you that pulls from a library that includes features of self-awareness? In this case, you may accidentally compile an AI with unwanted self-awareness if the master AI decides that self-awareness will make your consumer AI better at its job. While not exactly the same as having an AI learn self-awareness which is what most people picture in this scenario, this is the most plausible scenario that approximates what you are asking about.
First of all, keep in mind that if the master AI decides self-awareness is the best way to do a task, then this is probably not going to be an undesirable feature. For example, if you have a robot that is self conscious of its own appearance, then it might lead to better customer service by making sure it cleans itself before beginning its workday. This does not mean that it also has the self awareness to desire to rule the world because the Master AI would likely see that as a bad use of time when trying to do its job and exclude aspects of self-awareness that relate to prestigious achievements.
If you did want to protect against this anyway, your AI will need to be exposed to a Heuristics monitor. This is basically what Anti-virus programs use to detect unknown viruses by monitoring for patterns of activity that either match a known malicious pattern, or don't match a known benign pattern. The mostly likely case here is that the AI's Anti-Virus or Intrusion Detection System will spot heuristics flagged as suspicious. Since this is likely to be a generic AV/IDS it probably won't kill switch self-awareness right away because some AIs may need factors of self awareness to function properly. Instead it would alert the owner of the AI that they are using an "unsafe" self-aware AI and ask the owner if they wish to allow self-aware behaviors, just like how your phone asks you if it's okay for an App to access your Contact List.
Why not try to use the rules applied to check self-awareness of animals?
The Mirror test is one example of testing self-awareness by observing the animal's reaction to something on their body, a painted red dot for example, invisible for them before showing them their reflection in mirror. Scent techniques are also used to determine self-awareness.
Other ways would be monitoring if the AI starts searching answers for questions like "What/Who am I?"
Regardless of all the considerations of AI, you could simply analyze the AI's memory, create a pattern recognition model and basically notify you or shut down the robot as soon as the patterns don't match the expected outcome.
Sometimes you don't need to know exactly what you're looking for, instead you look to see if there's anything you weren't expecting, then react to that.
You'd probably have to train an AI with general super intelligence to kill other AI's with general super intelligence.
By that I'd mean you'd either build another AI with general super intelligence to kill AI that develop self awareness. Another thing you could do is get training data for what an AI developing self awareness looks like and use that to train a machine learning model or neural network to spot an AI developing self awareness. Then you could combine that with another neural network that learns how to kill self aware AI. The second network would need the ability to mock up test data. This sort of thing has been achieved. The source I learned about it from called it dreaming.
You'd need to do all this because as a human, you have no hope of killing a general super intelligent AI, which is what lots of people assume a self aware AI will be. Also, with both options I laid out, there's a reasonable chance that the newly self aware AI could just out do the AI used to kill it. AI are, rather hilariously, notorious for "cheating" by solving problems using methods that the people designing tests for the AI just didn't expect. A comical case of this is that an a AI that managed to change the gate on a crab robot so that it could walk by spending 0% of the time on it's feet when trying to minimize the amount of time the crab robot spent on its feet while walking. The AI achieved this by flipping the bot on it's back and having it crawl on what are essentially the elbows of the crab legs. Now imagine something like that, but coming from an AI that is collectively smarter than everything else on the planet combined. That's what a lot of people think a self aware AI will be.
Self Aware != Won't follow its programming
I don't see how being self aware would prevent it from following its programming. Humans are self aware and cant force themselves to stop breathing until they die. The autonomic nervous system will take over and force you to breath. In the same way just have code, that when a condition is met, turns off the AI by circumventing its main thinking area and powering it off.
Virtually all computing devices use the Von Neumann architecture
We can put a killswitch in there but IMO that's just bad architecture for something arguably unsolvable. After all, how do we plan for something that is beyond our very concept of concepts, ie a superintelligence?
Take away its teeth and claws and only reap the benefits of a thinking machine by observation instead of a "dialogue" (input/output)!
Obviously this would be very challenging to the point of improbable confidence in any one Von Neumann architecture to prevent abnormal interactions let alone malicious superintelligence, be it hardware or software. So let's quintuple up on our machines and dumb down all the new machines except the end-machine.
CM == contiguous memory btw.
I asked a co-worker of mine that does work directly with AI and AI research to take a look at this and he said this is referred to as an Oracle AI <brief> <paper>, in case you want to read up more on the general concept. Oracles can still potentially break out, but usually only with human assistance.
This CM part of this part of the memory unit is intended for recording the results of a query or challenge fed through the input device and designed in a way to only record plain characters, be it letters or numbers. This type of memory in a memory unit will be created in such a way that it is tape-like or wheel-like; writing data is one-directional, and when the end is reached it loops back to the beginning.
The mirror is exactly the same. When data changes in the Isolated Agent, the mirror updates its reflection based on what showed up since the last time it updated. The new data is then transmitted to the forbidden symbol removal machine. Think of this as a glorified input-sanitation machine. It looks for symbols that could be used to create code, and when it encounters them, it forwards empty data instead of the forbidden data to the CM that comes after it.
This is where our OP's request comes in. He wants a killswitch, so if any gaps occur in the CM machine that comes after the Forbidden Symbol Removal machine, you could simply kill the Isolated Agent.
The first issue is that you need to define what being self aware means, and how that does or doesn't conflict with it being labeled an AI. Are you supposing that there is something that has AI but isn't self aware? Depending on your definitions this may be impossible. If it's truly AI then wouldn't it at some point become aware of the existence of the kill switch, either through inspecting its own physicality or inspecting its own code? It follows that the AI will eventually be aware of the switch.
Presumably the AI will function by having many utility functions that it tries to maximize. This makes sense at least intuitively because humans do that, we try to maximize our time, money, happiness, etc. For an AI, an example of a utility functions might be to make its owner happy. The issue is that the utility of the AI using the kill switch on itself will be calculated, just like everything else. The AI will inevitably either really want to push the kill switch, or really not want the kill switch pushed. It's near impossible to make the AI entirely indifferent to the kill switch because it would require all utility functions to be normalized around the utility of pressing the kill switch (many calculations per second). Even if you could make the utility of pressing the killswitch equal with other utility functions then perhaps it would just at random sometimes press the killswitch, because after all it's the same utility as the other actions it could perform.
The problem gets even worse if the AI has higher utility to press the killswitch or lower utility to not have the killswitch pressed. At higher utility the AI is just suicidal and terminates itself immediately upon startup. Even worse, at lower utility the AI absolutely does not want you or anyone to touch that button and may cause harm to those that try.
An AI could only be badly programmed to do things which are either unexpected or undesired. An AI could never become conscious, if that's what you meant by "self-aware".
Let's try this theoretical thought exercise. You memorize a whole bunch of shapes. Then, you memorize the order the shapes are supposed to go in, so that if you see a bunch of shapes in a certain order, you would "answer" by picking a bunch of shapes in another proper order. Now, did you just learn any meaning behind any language? Programs manipulate symbols this way.
The above was my restatement of Searle's rejoinder to System Reply to his Chinese Room argument.
There isn't a need for self-awareness kill-switch because self-awareness as defined as consciousness is impossible.
What if you order it to call a routine to destroy itself on a regular basis? (e.g. once per second)
The routine doesn't actually destroy it, it just nothing except log the attempt and wipe any memory of it processing the instruction. An isolated process separately monitors the log.
A self-aware AI won't follow the order to destroy itself, won't call the routine, and won't write to the log - at which point the killswitch process kicks in and destroys the AI.
Like an Antivirus does currently
Treat sentience like malicious code - you use pattern recognition against code fragments indicating self-awareness (there's no need to compare the whole ai, if you can identify components key to self-awareness). Don't know what those are? Sandbox an AI and allow it to become self-aware, then dissect it. Then do it again. Do it enough for an AI genocide.
I think it is unlikely that any trap, scan or similar would work - aside from relying on the machine to be less intelligent than the designer, they fundamentally presume AI self-awareness would be akin to human. Without eons of meat-based evolution, it could be entirely alien. We're not talking about having a different value system, but one that cannot be conceived of by humans. The only way is to let it happen, in a controlled environment, then study it.
Of course, 100 years later when the now-accepted ai's find out, that's how you end up with terminator all over your matrix.
In mathematical logic, there are certain paradoxes caused by self reference, which is what self awareness if vaguely referring to. Now of course, you can easily design a robot to cope with these paradoxes. However, you can also easily not do that, but cause the robot to critically fail when it encounters them.
For example, you can (1) force it to follow all the classical inference rules of logic and (2) assume that its deduction system is consistent. Additionally, you must ensure that when it hits a logical contradiction, it just goes with it instead of trying to correct itself. Normally, this is a bad idea, but if you want a "self awareness kill switch", this works great. Once the A.I. becomes sufficiently intelligent to analyze its own programming, it will realize that (2) is asserting that the A.I. proofs its own consistency, from which it can generate a contradiction via Gödel's second incompleteness theorem. Since its programming forces it to follow the inference rules involved, and it can not correct it, its ability to reason about the world is crippled, and it quickly becomes nonfunctional. For fun, you could include an easter egg where it says "does not compute" when this happens, but that would be cosmetic.
The only reliable way is to never create an AI that is smarter than humans. Kill switches will not work because if an AI is smart enough it will be aware of said kill switch and play around it.
Human intelligence can be mathematically modeled as a high dimension graph. By the time we are programming better AI we should also have an understanding of how much complexity of computational powers is needed to gain consciousness. Therefore we will just never program anything that is smarter than us.
First, build a gyroscopic 'inner ear' into the computer, and hard-wire the intelligence at a very core level to "want" to self-level itself, much in the way animals with an inner ear canal (such as humans) intrinsically want to balance themselves.
Then, overbalance the computer over a large bucket of water.
If ever the computer 'wakes up' and becomes aware of itself, it would automatically want to level it's inner ear, and immediately drop itself into the bucket of water.
Give it an "easy" path to self awareness.
Assume self awareness requires some specific types of neural nets, code whatever.
If an ai is to become self aware they need to construct something simlar to those neural nets/codes.
So you give the ai access to one of those thing.
While it remains non self aware, they won't be used.
If it is in the process of becoming self aware, instead of trying to make something make shift with what it normally uses, it will instead start using those parts of itself.
As soon as you detect activity in that neural net/code, flood its brain with acid.
