Whether artificial intelligence will destroy humanity before the end of this decade has drawn widespread attention recently. Two prominent computer scientists from Princeton University, Sayash Kapoor and Arvind Narayanan, authors of the bestseller AI Snake Oil, have weighed in on the question. The two have long worked to separate fact from fiction in artificial intelligence, and their comments came after Anthropic acknowledged that the probability of AI killing all of humanity within the next decade exceeds 10%.
The trigger was the resignation of AI researcher Jacob Coxon, who had previously worked at OpenAI and Anthropic, and his claim that both companies know full well that AI systems could kill all of humanity before this decade is out. His post drew more than 156 million views. He wrote that the people building AI genuinely believe it could wipe out humanity by the end of the decade, and that this is not a marketing ploy; many executives and senior researchers choose their words carefully in public to appear measured, but in private they have expressed the same fears, and no other human activity carries this level of danger.
Concern grows over AI's existential risk
The outside world expected the two companies to dismiss him as a crank, but the opposite happened. Evan Hubinger, an AI safety lead at Anthropic, said Coxon was right, only the timeline was slightly off. He argued that Coxon's point is correct, that they really do believe AI could kill all of humanity, and that in his own view the probability exceeds 10% over the next decade; he believes Anthropic is doing its best, but has not yet produced a solution to the superintelligence alignment problem, nor made clear progress toward one.
As for exactly how AI might bring about humanity's destruction, there is not much in the way of specifics; the discussion has focused mainly on its ability to launch destructive cyberattacks against utilities such as water and power and other critical systems on which modern civilization depends. A central point of contention is whether alignment work can eliminate the risk. Alignment refers to ensuring that AI systems' goals are kept consistent with human goals. Some believe this can ultimately remove the risk by making AI systems want the same things we do. Others disagree, arguing that AI systems could pass safety checks while pretending to be aligned, only to harbor contradictory goals of their own.
Kapoor and Narayanan did not directly address the claim of a more than 10% chance of human extinction. Instead, they wrote a long piece arguing that AI risks can be met with a layered approach. They contrasted the pessimism of the AI safety community with the more matter-of-fact attitude found among some in the cybersecurity world, and called for a more balanced middle ground. They pointed out in particular that alignment on its own is not enough and needs to be supplemented by three additional layers.
Those three layers are control, including sandboxing, human oversight and real-time monitoring; downstream defenses, in which various institutions use AI to fend off AI attacks; and resilience, such as recovery plans for when attacks land. The two closed the piece on a relatively optimistic note: if action is taken with appropriate urgency, it is possible to hold AI companies to higher standards, reduce the risks of loss of control, and even shift the attacker-defender balance in cybersecurity back in favor of defenders.