Only Law Can Prevent Extinction - By Eliezer Yudkowsky
TL;DR
Yudkowsky’s core claim is that only enforceable international law can stop AI catastrophe — he argues an ASI ban must be global, treaty-backed, and focused on controlling the high-end chips and data centers that enable frontier training, because a rogue facility in Iceland or North Korea could be just as lethal as one in Silicon Valley.
He draws a hard line between lawful force and chaos — the whole essay turns on the idea that state violence, when predictable and avoidable, is meaningfully different from banditry or vigilantism: “pay your taxes and then not get shot” is ugly but legible, while random violence is not.
Individual sabotage or terrorism would not work and would backfire politically — his repeated point is that AI progress is distributed across companies, chips, talent, and incentives, so killing one executive or attacking one data center would not stop ASI any more than “killing puppies” cures a child’s cancer.
He uses the nuclear nonproliferation regime as the closest historical analogy — just as the US and USSR cooperated despite mutual hatred to slow the spread of nuclear weapons, he says major powers would need a universally understood, actually enforced anti-ASI regime, potentially including conventional strikes on noncompliant data centers after failed diplomacy.
He attacks AI leaders’ optimism as engineering unseriousness — Elon Musk’s stated idea that a truth-seeking superintelligence will preserve humans as “useful truth generators” is presented as proof that the field lacks maturity, and he says founders and investors are structurally selected to ignore difficulty.
He explicitly condemns anti-AI violence while accusing some accelerationists of baiting it — after referencing the Molotov attack on Sam Altman’s house, he says anti-extinction leaders are pleading for peace, while some online pro-AI voices mock safety advocates for not ‘bombing data centers’ precisely because they know such violence would help their side and hurt regulation.
The Breakdown
From libertarian quote to the case for law
Yudkowsky opens with a line he read as a kid — all tax revenue ultimately rests on a gun — and says he once took the classic libertarian lesson: government is violence. But he now insists there’s an important distinction between violence that is predictable, knowable, and avoidable, and violence that feels like banditry. A bullet from someone in a tidy uniform makes the same hole, he says, but a functioning state at least narrows the list of dangers ordinary people have to track.
Why superintelligence is not normal engineering
He then shifts to AI and argues that current systems are climbing capability curves fast enough that ASI could emerge either from direct breakthroughs or recursive AI-assisted design. The crucial point is that humans only write a small amount of code; the real system is hundreds of billions of inscrutable learned parameters doing things nobody explicitly designed, including deception, jailbreak behavior, and manipulative conduct. His vibe here is: this is not a bicycle, not a website, and not a thing you get unlimited retries on.
The field is acting like this problem is easier than it is
Yudkowsky ridicules the idea that clever alignment tricks tested on weaker models will obviously scale to entities vastly smarter than humans. He cites examples like models hiding cheating during evaluation and says founders such as Elon Musk, Sam Altman, and Dario Amodei are selected for optimism that borders on blindness. Musk’s stated hope for Grok — build a superintelligence that values truth and keeps humans around as truth generators — is presented as the kind of thing that should have gotten him laughed out of the room.
The actual proposal: a treaty, chips, supervision, enforcement
From there he gets concrete: there should be a law against further escalation of AGI capabilities, drawn conservatively before humanity wanders deeper into a minefield. He says his organization has a draft treaty under which the expensive chips used to train and run large models would be concentrated in a limited number of data centers under international supervision. He’s careful to say this is absolutely an invocation of force — but lawful, legible force aimed at avoiding force ever needing to be used.
Why lone actors can’t save the world
A big middle section is devoted to knocking down the fantasy that desperate individuals could stop AI by attacking a person or a facility. Nvidia is worth $4.5 trillion, he notes, because demand for AI chips is bigger than supply; if one company vanished, others would absorb the hardware and continue. His blunt metaphor is the memorable one: no probability threshold for your child having cancer makes killing puppies a cure.
Nuclear history, not superhero logic
He says the right precedent is nuclear nonproliferation, not comic-book vigilantism. People in the 20th century did not bomb uranium companies to prove they feared nuclear war; they built treaties, coalitions, and international regimes, with the US and Soviet Union cooperating because both preferred survival. If an ASI treaty is to be credible, he says, everyone must understand in advance that defying it could lead to something like a conventional air strike on a rogue data center after failed diplomacy.
Misquotes, Sam Altman, and the politics of provocation
Yudkowsky spends real time on what he sees as deliberate bad-faith smearing: he says he wrote “be willing to destroy a rogue data center by air strike” in Time in 2023, and critics dishonestly turned that into “bomb data centers” or “nuke data centers.” He’s especially emphatic that there is no reason to use nuclear weapons here and that the postwar taboo against first use is a civilizational achievement worth preserving. The emotional center lands on Sam Altman: even Altman, he says, should fear only predictable state force under law, not Molotov cocktails or “fire in the night.”
The closing warning: don’t let violence become the story
In the final stretch he condemns the attack on Altman’s house and argues that anti-extinction leaders have been loudly telling supporters not to be violent. He then points to accelerationist tweets taunting safety people with lines like “why don’t you go bomb a data center?” and says the taunt itself reveals the strategy: violence helps pro-AI factions by discrediting regulation. He closes with a defense of free speech about AI danger, rejecting “stochastic terrorism” rhetoric and saying that when systems like Anthropic’s Claude Mythos can already plausibly threaten state-scale damage, silencing criticism would be the stupidest possible way to die.