1 hour ago
11
With Jacob Coxon’s resignation from Anthropic, we have reached the AI risk tipping point. Millions of people are finally coming to understand what experts have known for years: AI companies have been gambling with all of our lives, and the odds are not good.
In response, the Anthropic CEO, Dario Amodei, has introduced a proposal for “pacing the frontier”, endorsed by Sam Altman and Elon Musk. Can we breathe a sigh of relief? Are we about to step back from the edge of extinction?
Amodei envisions a slowdown of one to two years, resulting in “profound progress” on technical safety measures. But that’s not what humanity needs right now. What we need is a plan in which we’re confident AI isn’t going to kill us, and this isn’t it.
The stark reality is this: we don’t know how these AI systems work. We don’t know how to prevent them from misbehaving. We don’t know how to stay in control once they’re smarter than us. We don’t know how to make sure they’re not playing nice or playing dumb to trick our safety tests. We can’t count on solving these problems with another one or two years of research.
I know because I have been researching such problems for more than a decade, including as an AI professor at both the University of Cambridge and the University of Montreal. If we want to reduce the risks from AI to an acceptable level, what we actually need is an immediate, indefinite, international moratorium on frontier AI development.
Is this really possible? It’s a legitimate question, and geopolitics is the crux: how can countries such as the US and China verify each others’ compliance? Donald Trump recently slammed AI regulation, citing the US’s lead in the AI arms race against Beijing.
But a simple solution is available: de-commission the advanced computer chips necessary to build these AI models. The massive – and already massively unpopular – datacenter build-out could be reversed. The US and China could take turns decommissioning data centers and the chips they contain. If all parties knew that nobody had the means of building more powerful AI, verification would be simple.
To make such an arrangement durable, we could additionally dismantle the supply chain for producing advanced AI chips, which is extremely concentrated and relies on some of the most advanced technology in the world: extreme ultraviolet (EUV) and deep ultraviolet (DUV) lithography. Any attempt to operate a covert lithography operation at the scale required for frontier AI development would almost certainly be noticed by other nations’ intelligence services.
This plan has benefits over alternative approaches that rely on restricting how computer chips are used. Amodei calls pausing “unlikely”, but an agreement to decommission AI infrastructure would actually be easier to verify than Amodei’s proposals for global coordination, which would require increasing levels of surveillance as computing hardware continues to advance and proliferate.
The economic costs of this plan would be substantial, but given the stakes, unless we have a better plan that actually saves our lives, they are clearly worth it. During the Covid-19 pandemic, governments shut down significant parts of society and the economy to save lives. We should be willing to go to much greater lengths to prevent the extinction of our species. I’m actively researching “softer” versions of the proposal that would involve creating computer chips that can be used to run AI models, but not to build more powerful ones. And other researchers have also developed proposals for an indefinite moratorium.
Amodei’s essay, sadly, does not consider these pause proposals. His plan is ultimately still to have AIs build smarter and smarter AIs with less human involvement – “recursive self-improvement”, a project designed to render humans obsolete.
What should we actually expect to come of Amodei’s plan? Step one would “embed” external evaluators from organizations such as METR, which was invited to investigate the OpenAI-Hugging Face scandal. Big tech CEOs may call this “radical”, but let’s be clear: what they’re actually talking about is more self-regulation.
Three years ago, I helped found the UK AI Security Institute – another leading evaluator. I quit after it was clear the ecosystem was getting the burden of proof formula backwards. Today, AI is still de facto “safe-until-proven dangerous”.
The safety standards Amodei suggests are guaranteed to fall far short of those in other safety-critical industries like aviation. We simply don’t know how to do recursive self-improvement safely.
So let’s stop. Let’s stop making AI we can’t control, and let’s stop listening to the AI industry about how it wants to be regulated. Proposals for a pause are being put forth in both the US and UK legislatures. The Chinese president, Xi Jinping, is coming to Washington to discuss AI, and Trump has an opportunity to make the deal of the century if they can work out how to stop the AI race.
Let’s demand a plan that makes us safe, not just safer.
-
David Krueger is an assistant professor in Robust, Reasoning and Responsible AI at the University of Montreal. He is also the founder of Evitable, a non-profit that educates the public about the risks of artificial intelligence
