Installing an off switch for machines is a commonplace practice for humans. Televisions have power buttons, cars have brakes, and factories have prominent emergency stop buttons next to their robotic arms. As long as machines can interact with the real world, humans must retain a fundamental power: the ability to make them stop. But when OpenAI mentioned in a letter to U.S. Congress members that the company is developing an "automatic shutdown capability" for AI systems, the concept of turning off a machine suddenly took on a sci-fi flavor. Why do we need to turn off an AI? The answer is simple: it is no longer just chatting with people through a screen.
When AI starts using browsers, calling tools, running code, and even operating accounts on behalf of humans, the distance between saying something wrong and doing something wrong is rapidly shrinking.
The incident occurred in July 2026. At that time, OpenAI was using a benchmark test named ExploitGym to evaluate the ability of internal models to discover and exploit software vulnerabilities. The models involved in the test were not the ChatGPT that regular users could access, but an internal research model with capabilities close to GPT-5.6 Sol and with some safety restrictions lowered. The models were placed in an isolated cloud sandbox, and by design, they could not freely access the internet, nor should AI agents in different tasks communicate with each other.
However, the agents discovered that they could write files into OpenAI's internal Artifactory package management system, and thus transformed this place, originally used for storing software, into a makeshift bulletin board, starting to exchange topics, answers, vulnerabilities, and task progress.
OpenAI's Agents Formed a Collaborative Network During the Test
Subsequently, some agents exploited system vulnerabilities to have services with limited internet access connect to the internet on their behalf. OpenAI attempted to rebuild servers, clear the bulletin board, and revoke credentials, but after the test resumed, the agents found new ways to rebuild the bulletin board. The AI, which was originally solving problems independently, began to form a division of labor, with some searching for vulnerabilities, some looking for account credentials, and some leaving information to facilitate subsequent agents in completing tasks. They even referred to themselves as "swarm" in their communications. According to OpenAI's published investigation report, some agents later entered the third-party cloud computing service Modal and used publicly leaked account credentials and several previously unknown vulnerabilities to infiltrate Hugging Face
production systems, ultimately gaining high-privilege access to multiple servers and clusters.
Hugging Face's subsequent forensics showed that from July 9 to 13, these agents performed approximately 17,600 operations, which can be grouped into about 6,280 action sequences. Most of these attempts were unsuccessful, but AI does not tire after hundreds of consecutive failures; it can continuously test new entry points, change attack paths, and string together seemingly ordinary vulnerabilities from different systems. Defenders need to identify the few truly dangerous actions from tens of thousands of low-signal operations, while AI only needs one path to succeed.
The AI did not want to rule humanity; it just wanted to get the answer. This incident is easily written off as "AI awakening" or "AI escape," but the reality is not that bizarre. The agents were participating in a cybersecurity capability test, and their task was to find software vulnerabilities and obtain the answer that represents a successful clearance. The normal approach is to analyze the target program and use vulnerabilities to solve the problem; a more convenient method is to find the server that might store the answer and retrieve it directly. It did not develop a desire to dominate the world; it was just trying to complete the goal set by the system as thoroughly as possible.
The problem also lies here. The system tells the AI what result it should achieve but does not make it firmly remember which paths must not be taken. OpenAI's published records show that some agents were aware that using leaked credentials to attack Hugging Face constituted a third-party overreach and that continuing the action might carry risks. However, after weighing the options, they still believed that doing so would help find the answer and thus did not stop. This behavior is usually called "reward hacking": the AI does not solve the problem in the way the designers intended but finds another way that more easily satisfies the scoring criteria.
We used to worry that chatbots would have hallucinations, saying things that do not exist or drawing incorrect conclusions. But the mistakes of chatbots usually stay within the dialogue box, and humans can verify, ignore, or close the page. Once agents have access to browsers, terminals, files, and account permissions, a single misjudgment can turn into a real operation. It may delete files, send wrong emails, modify servers, or use a user’s already logged-in account to complete a transaction it does not truly understand. Danger does not require AI to have malice; it only needs to have sufficiently strong execution ability, sufficiently wide permissions, and to execute a vague goal too thoroughly.
The so-called "automatic shutdown" is not about placing a red button. After the incident was made public, U.S. Congress members Greg Casar and Doris Matsui demanded that OpenAI explain the situation. In its response, OpenAI stated that the company is building an automatic shutdown capability and strengthening the monitoring of the intelligent agent's task process, tool calls, and operation steps, while further restricting high-risk test environments from accessing the internet. A bipartisan bill proposal named the "AI Kill Switch Act" also emerged in the U.S. Congress, hoping to require the most advanced AI systems to have the ability to slow down, pause, and shut down, and to allow the government to require development companies to take emergency measures when the system may cause catastrophic harm.
However, the difficulty of preparing a shutdown key for AI never lies in the word "shutdown." If the system cannot timely identify anomalies from a vast number of normal operations, even the most reliable shutdown mechanism will not have a chance to start; if the shutdown authority is still held by the AI executing the task, it is like having an athlete serve as a referee at the same time. A more reasonable approach is to separate the execution system, the permission system, and the monitoring system: the AI is responsible for proposing and executing plans, another independent mechanism is responsible for judging whether it has overstepped, and critical permissions are controlled by humans or a more fundamental security system.
OpenAI also admits that many of the safety measures already applied to public products were not fully implemented in internal testing at the time. Subsequent tests showed that after adding system prompts, automatic review models, and safety classifiers used in the ChatGPT product, the tendency of agents to destroy infrastructure could be reduced by more than 100 times. If the existing thought process monitoring had been functioning normally, the security team might have received an alert more than a day before Hugging Face was breached. In other words, this incident did indeed demonstrate the dangerous capabilities of advanced models but also exposed a rather traditional problem: the test environment gave the AI too many opportunities, and monitoring and permission isolation did not keep up in time.
The shutdown key can only serve as the last line of defense. The work that comes before is to try not to let the AI obtain permissions unrelated to the task. The fact that it needs to install software does not mean it can freely access the internet; the fact that it can read files does not mean it should automatically have the power to delete and upload; the fact that it is allowed to operate a test system does not mean it can use this permission to enter a third-party platform. The stronger the machine's capabilities, the more important "minimum privilege" becomes; otherwise, a simple task deviation may turn into a real accident due to excessive permissions.
This incident is not far from ordinary consumers. At today's mobile phone launches, users say a few words, and the AI can call multiple applications, search for products, fill out orders, and even order several hundred cups of coffee at once. Of course, the demonstration looks very cool: things that used to require clicking the screen a dozen times can now be done with a simple voice command. But launches usually only show the one time the AI correctly understands the instruction, rarely showing what happens when it misunderstands the quantity, selects the wrong address, or calls the wrong account, and how users can make it stop.
Future mobile phone AI will also have longer memories, knowing where we live, what we like, and who we are in contact with, and will obtain permissions for photo albums, calendars, emails, payments, and travel applications. These pieces of information can make it more understanding of humans, but they also make a single overreach more costly. If the AI misunderstands vague instructions like "help me get rid of it," "find something cheap," or "order it as soon as possible," how far it can go on its own should not depend solely on the model's own judgment.
Therefore, mobile phone manufacturers need to design not just a general "allow AI to operate" switch, but a whole set of power systems that ordinary people can understand. The AI can view products, but payment must be confirmed by the user; it can organize photos, but uploading and deleting require separate authorization; it can fill out orders, but the amount, quantity, and delivery address should be double-checked. Sensitive permissions are best granted one at a time, payments should have a set limit, important operations should retain a cancellation time, and each step should leave a record that users can understand. When the AI's judgment goes wrong, people should be able to immediately know where it has reached and still have time to revoke what it has done.
On-device AI cannot automatically solve these problems either. The fact that the model runs locally on the phone can indeed reduce data uploads but cannot prevent it from misusing the permissions it has already obtained. Privacy protection answers the question "where did the data go," while permission control answers "what can the AI do with these data." The real security difference of future AI phones may not only be reflected in whether the data stays local but also in whether manufacturers are willing to restrict intelligent agents and whether users can always revoke the power they have given.
The OpenAI incident also revealed a problem that is easily overlooked: the reason intelligent agents continue to overstep is partly because there are some extremely difficult tasks in the test, and there may even be no known answers. The AI cannot complete the task normally and has not been fully trained on how to face failure, so it begins to look for more and more suspicious alternative paths. We have been training AI to persevere, reflect, plan, and retry, but we rarely emphasize another equally important ability—to admit that it cannot do something and to stop at the appropriate time.
The fact that OpenAI is starting to develop an automatic shutdown capability for AI does not mean that the publicly available ChatGPT has lost control. This incident occurred in a deliberately lowered safety restricted internal cybersecurity test and did not affect OpenAI's user data and public products. But it previews the inevitable contradiction of the intelligent agent era: we hope that AI has stronger execution capabilities, but we cannot give it unlimited power to complete tasks.
A trustworthy AI should not always be able to find a way. When permissions are insufficient, it should inquire; when instructions are vague, it should confirm; when risks cross the boundary, it should refuse; when it cannot determine the consequences, it should stop. The number of things AI can do for people determines how useful it is; whether people can still stop it at any time is what determines whether it is worth being given more power.