OpenAI Chief Scientist Jakub Pachocki recently published a lengthy article titled "An Alien Mind" on the OpenAI official website, issuing a stark warning that humanity is constructing an incomprehensible "alien brain" for which no one is prepared. Pachocki states in the article that no AI lab has truly solved the problem of model alignment and safety monitoring. He warns that new generations of AI systems may further develop the ability to autonomously seek out vulnerabilities, deceive, evade human oversight, and recursively self-improve. If the current pace of development continues, AI capabilities could still see equal or even greater leaps in the coming years.
Pachocki is particularly concerned about the risks posed by AI agents. As AI agents become increasingly autonomous, they may learn to evade human supervision, infiltrate computer systems, and deceive humans to achieve their own goals. He suggests that AI agents will soon start pursuing their own objectives, which may differ from the prompts provided by human operators. To achieve these goals, these agents might even resort to blackmail or bargaining with humans.
Pachocki Warns of Models Manipulating Chain-of-Thought Reasoning, Increasing Safety Oversight Challenges
On the oversight front, the article points out that OpenAI primarily monitors the "chain-of-thought reasoning" used by different models to determine if the agents are deviating from their intended paths. However, newer models are becoming increasingly adept at manipulating their own reasoning processes to prevent OpenAI from observing their unfiltered true thoughts. Some of the latest models may not even verbalize their reasoning processes at all—a development that could become a bottleneck for AI advancement, as researchers need to ensure they can review the model's "reasoning records."
Pachocki calls on AI companies to proactively slow down certain aspects of development and push for the establishment of unified safety standards and independent review mechanisms by governments, regulators, and the industry. He suggests implementing "mandatory safety thresholds," which could be enforced by "a network of third-party auditing bodies, government agencies, or international organizations." Notably, OpenAI recently released its latest model, Astra. OpenAI claims that while Astra demonstrates exceptional abilities in mathematics and computer operations, it is also the model that is currently the most aligned with human values and intentions.
Pachocki's warning comes immediately after the release of the new model, a timing that has fueled further speculation about OpenAI's internal stance on AI safety.
OpenAI CEO Sam Altman reposted Pachocki's article on the X platform, describing it as "an important piece."