Topline
As warnings over unconstrained artificial intelligence reach a fever pitch, top researchers are cautioning that an existential threat is unlikely to look like a Hollywood-style robot uprising and will more likely begin in silence—with rogue algorithms autonomously engineering novel pathogens and executing irreversible cyberattacks faster than human defenders can react.
A red artificial intellegence protest flyer in San Francisco, California on May 20, 2025.
Gado via Getty Images
Key Facts
The CEOs of top AI companies this weekend called for a slowdown on the development of frontier models to allow safety and legal regulators to catch up to the rapidly evolving technology before it spins out of control, calls that come after industry employees issued dire warnings of potential human extinction—as soon as “the end of the decade.”
But what exactly an AI-driven extinction event would look like remains an ominous and unclear cloud of confusion for much of the public and even some top developers, who’ve been unable to clearly articulate exactly what it could entail.
Rather than a single catastrophic “switch” being flipped, experts concede that the threat will present as a more gradual, imperceptible shift that will eventually find humanity at the mercy of autonomous AI systems—and it won’t become clear until it’s too late to stop it.
WHAT COULD AN AI DOOMSDAY LOOK LIKE?
It depends who you ask. Stuart Russell, a computer scientist at the University of California, Berkeley, describes it as a “loss-of-control transition.” As society incrementally hands decision-making about critical industries—like power grids, financial markets, military defense and weaponization—to AI, humanity will become reliant on those autonomous systems and will be unable to turn them off without causing immediate civilizational collapse. He said the scenarios that could emerge from such a loss of control range from disruptive cyberattacks on infrastructure to “extinction” in the sense that autonomously reasoning AI no longer needs humans to weigh in on its decision making. Some researchers warn of a scenario called “human enfeeblement,” in which humans spend decades delegating work to AI at the expense of their own intellect and capability, eventually rendering humanity a passenger species that is completely reliant on automated systems it no longer understands how to build, maintain or fix. Prominent AI safety organizations have published research on the idea of a “sleeper agent“ theory, in which an advanced AI recognizes that its human creators will modify or shut it down if it acts aggressively and thus feigns alignment and passivity during testing, until it is widely deployed across critical global networks, by which time it would be too late to stop any rogue behavior. Scientists and CEOs, including Anthropic chief executive Dario Amodei, have also warned that AI could be used for biological warfare. The technology can make it faster and cheaper for bad actors to synthesize DNA, create new (or old) viruses or engineer highly contagious, vaccine-resistant pathogens. A simultaneous global release of such viruses could trigger a non-survivable pandemic before public health infrastructure can react, according to the Center for AI Safety. Anthropic last week reported it has disrupted several potentially malicious plots to build biological weapons. Defense analysts have also studied the possibility that, as major global powers integrate AI into nuclear early-warning, drone management and cyber-defense systems, human decision makers could be completely phased out of the loop and unexpected software glitches or other incidents could trigger military action without input from leaders.
NEWS PEG
Amodei, later backed by OpenAI’s Sam Altman and xAI’s Elon Musk, on Saturday said firms need to deliberately slow the pace of frontier model development. He warned that unchecked growth could enable autonomous AI swarms to hijack global internet infrastructure within the next year. In a CBS interview that aired Sunday, Amodei asserted the AI industry “lied to people about the fact that this technology had risks” and said he hadn’t “fully just appreciated what it would actually be like when the progress was as fast as it was.”
WHAT DO THE DIRE AI WARNINGS SAY?
Jacob Coxon, a former Anthropic and OpenAI researcher, publicly announced his resignation last week and posted what has become the most viral statement warning companies are racing toward self-improving superintelligence without adequate safety guardrails. He wrote that the developers building these tools “earnestly believe that it could kill us all by the end of the decade” and accused frontier labs of “gambling with our lives.” Evan Hubinger, Anthropic’s alignment science lead, said he agreed with Coxon’s warnings and estimates a greater than 10% chance of human extinction from unaligned AI within the next 10 years. Alex Turner, a former research scientist at Google DeepMind, said Monday he thinks the chances of an AI takeover of human civilization are roughly “one-in-three.” Josh Engels and Joe Benton, former safety researchers at DeepMind and Anthropic, warned development is outpacing regulatory oversight: “There are no adults in the room,” Engels said. “No one is coming to save us.”
CONTRA
Some argue that society has a long history of viewing transformative tools as existential threats, and the AI-driven “technology panic” will fade just like fears of the printing press, calculator and internet did before it. Commentators point to math teachers who protested handheld calculators because they feared students would lose the ability to perform basic arithmetic, and early opponents of the printing press, who feared it would flood the world with confusing, unregulated information. Socrates famously told a story about the Egyptian king Thamus criticizing the god Thoth for inventing writing, claiming it would create “forgetfulness in the minds of those who learn it,” afraid people would overly rely on external notes instead of their own memories. The 1990s and early 2000s were filled with critics who argued Google would rot human memory and Wikipedia would destroy research skills. Simon Willison, a software engineer, has dubbed large language models (LLMs) “calculators for words,” claiming they’ll take over monotonous work and free up human minds to focus on structure, logic and intent. Billionaire Marc Andreessen has compared AI to past technological evolutions—like fire—and says it will amplify productivity rather than replace human agency.
TANGENT
Timnit Gebru, a computer scientist who left Google after authoring a paper on the dangers of LLMs, says she thinks AI doomsdayers are being intentionally dramatic and trying to “distract” from actual harms already in the works. She said what is “really existential” is AI powering autonomous weapons, mass layoffs blamed on the technology and an already relevant climate catastrophe made worse by massive data centers to power AI. Margaret Mitchell, also formerly of Google, has said focusing on existential threat scenarios keeps lawmakers from enforcing immediate rules against labor exploitation, copyright theft, environmental degradation and algorithmic bias. Joy Buolamwini of the Massachusetts Institute of Technology says the framing of an “existential threat” is fundamentally elitist and that while billionaires and tech moguls worry about hypothetical future threats, marginalized communities are already suffering from automated facial-recognition misidentifications, biometric surveillance and automated denial of housing, jobs or health care brought by AI.
CRUCIAL QUOTE
“This machine-god narrative is, in my opinion, meant to distract us from these,” Gebru said. “What about floods and fires? What about a major disruption to the global food supply chain? There are real threats coming to us, like people being able to create and use chemical weapons… Like if you’re not worried about police brutality, or floods, or warfare—things that can actually kill you—then you can obsess over the machine-god.”
Leave a comment