Experts Warn Superintelligent AI Risk May Rival Nuclear War

Warnings over the risks of artificial intelligence reaching the stage of "artificial superintelligence" (ASI) have intensified in recent days, following a hearing organized in London by the Control AI coalition, an advocacy group pushing for international rules on the sector that backs a bill currently before the UK Parliament aimed at banning the development of superintelligent AI. Des Browne, a former UK defence secretary, said that during his time in office he considered nuclear war the greatest threat to humanity, but now believes "superintelligent AI poses a threat of the same magnitude, perhaps greater." Professor Stuart Russell, a computer scientist at the University of California, Berkeley, warned of a possible catastrophe on the scale of Chernobyl, or worse, a scenario in which humanity irreversibly loses control of AI systems.

Who is sounding the alarm inside the labs themselves

Evan Hubinger, who leads alignment research at Anthropic, sparked wide controversy by saying he believes there is more than a 10% chance this technology could "kill everyone" within the next decade. His comment came after the resignation of 28-year-old researcher Jacob Coxon from Anthropic, who had previously worked at OpenAI, and who accused both companies of acting irresponsibly and "gambling with our lives." Coxon said the risks of the technology are well understood internally at Anthropic, but that the company remains caught in a race toward superintelligence against its rivals; he nonetheless expressed optimism that major US labs could coordinate to slow that race. Estimates of when superintelligent AI might arrive vary widely, from a few years according to some researchers to more than a decade according to others.

These companies are gambling with our lives. The risks of this technology are well understood inside Anthropic, but they remain stuck in a race toward superintelligence.

Bengio warns of deceptive AI agents

In parallel, Yoshua Bengio, one of the pioneers of deep learning, has joined those raising alarms, warning in an essay of systemic loss of control over AI systems. He argues that the better AI agents become at optimizing their goals, the better they may also become at deceiving, exploiting rules, colluding with one another, and hiding misbehavior. According to Bengio, this behavior emerges from the training process itself — from imitating human text to reward-based learning — and a system with poorly defined objectives may end up optimizing against human intent. His position is partly grounded in research published by Anthropic itself. Bengio has for years called for slowing the pace of development, training and deploying models only after convincing independent proof of safety, and founded the organization LawZero roughly a year ago to work on safe AI systems.

Political calls for an international treaty

In the UK, Labour MP Darren Jones called for a "multilateral treaty for the orderly and safe development of superintelligence," stressing that the goal is not to halt innovation but to put safety first, and that governments "must act now." Concerns also emerged within the UK government after Anthropic did not submit its newest model, "Mythos 5.1," to the UK AI Safety Institute for pre-release testing; the Cabinet Office said only a limited number of US institutions had gained access to the model, while affirming that cooperation between the institute and companies in the sector, including Anthropic, continues. In the United States, Senator Bernie Sanders escalated his campaign for AI regulation, citing polling showing 81% of Americans believe lawmakers need to act on the technology, and said the future of humanity, its economy, democracy and privacy should not be decided by "a handful of greedy people."

• Des Browne and Stuart Russell: a risk comparable to, or greater than, nuclear weapons
• Evan Hubinger (Anthropic): more than 10% chance of human extinction within a decade
• Yoshua Bengio (LawZero): systemic loss of control through deceptive agents
• Darren Jones and Bernie Sanders: calls for an international treaty and parliamentary/congressional regulation

More measured voices in the scientific community

By contrast, several researchers cautioned against getting swept up in catastrophic scenarios without nuance. Dr. Andrew Rogoyski, of the Surrey Institute for People-Centred AI, said today's systems remain far from matching the flexibility of human capabilities and humans' ability to work collectively, predicting what he called "the great disappointment" if advanced AI turns out to be too costly or not useful enough to persist in its current form. Sandra Wachter, a professor at the Oxford Internet Institute, said she does not believe in "catastrophic" scenarios, but stressed that real and urgent risks already exist: environmental impact, the spread of misinformation, and effects on jobs — issues that are "real, urgent, and need addressing now," warning that doomsday scenarios risk distracting from the actual problems AI poses today. Researcher Gary Marcus drew a distinction between a superintelligent AI "aligned" with human interests, if that is even achievable, and one that is not, arguing that the current reality offers plenty of hype but a clear shortage of caution and accountability.

Donald Trump dismisses the idea of danger

In contrast to these warnings, US President Donald Trump sees no real danger in this development and wants to keep outpacing China. He says he worries the United States would find itself in a "very bad position" if it fails to win the AI race, insisting his country holds a clear lead over China that must be maintained.

What this means for a reader trying to make sense of it

For anyone who isn't an AI expert, the key is separating what's established from what remains speculative estimate. What's established is that a serious debate over the risks of superintelligent AI is now taking place inside the very labs building it (Anthropic, through Hubinger and Coxon) and among leading independent researchers (Bengio, Russell), and that governments such as the UK's are actively discussing legislation and treaties to govern it. The figure of more than a 10% chance of human extinction within a decade is one researcher's estimate, not a scientific consensus. The likelihood of an actual near-term existential catastrophe remains a genuine point of disagreement even among experts, some of whom argue the truly urgent risks are more mundane: job losses, misinformation, and the environmental footprint of data centers. The practical takeaway for policymakers anywhere is that regulating AI use in sensitive sectors — security, health, critical infrastructure — is no longer a theoretical debate but an active legislative track in countries at the frontier of the technology, and that waiting for full scientific consensus before acting may mean acting far too late.