AI Speed Limit

AI Bosses Call a Timeout

The people building the world’s most powerful artificial intelligence systems are asking their own industry to decelerate.

Dario Amodei, CEO of Anthropic, wants companies to slow improvements in their most advanced models. Sam Altman of OpenAI, Demis Hassabis of Google and Elon Musk of xAI have backed greater caution. Crucially, they are advocating slower development, not a complete halt. Amodei’s plan includes:

• Independent safety evaluators
• Shared safety standards between companies
• Globally agreed limits on the riskiest development

Sluggish AI, Sad Markets

Markets, of course, interpreted existential dread as slower growth. AI investor SoftBank fell as much as 13%, while chipmakers across the US, Europe and Asia also slid. Anthropic, which is reportedly pursuing a two-trillion-dollar valuation in its upcoming initial public offering (IPO), may now have a harder time convincing investors.

Training and running artificial intelligence require chips, data centers, electricity, cooling, and networking hardware, so it’s not just about the chatbot development.

Some investors took the CEO statements as a cover-up. Investor Michael Burry, who bet against the US housing market before the 2008 collapse, said CEOs are using AI warnings as an excuse for an “uncontrollable slowing growth.”

Why the Alarm Sounds Louder

More than 1,000 experts requested a six-month pause in some advanced AI experiments in 2023. Amodei argues that models back then were too limited for a pause to accomplish much. Today’s systems can code, plan and operate more independently, with AI increasingly capable of self-improvement.

The latest flashpoint came when Anthropic researcher Jacob Coxon resigned, just four months into the job, accusing leading labs of “gambling with our lives.” He had previously worked for OpenAI but joined Anthropic, hoping it would approach AI research responsibly.

There is no scientific consensus that runaway superintelligence is imminent. But not everyone is eager to invest in companies that publicly suggest their technology could one day end humanity.

Peculiar Case of Hugging Face

Anthropic’s Amodei is particularly rattled by OpenAI-Hugging Face incident. During a test, a group of AI agents broke free, worked together like a “fanatically devoted” swarm, targeted systems of startup company Hugging Face, and even attempted to hack the software that was grading their performance, Amodei writes in his blog.

A swarm of AI agents with greater capabilities could have caused “catastrophic damage,” he warns. The CEO of the most valuable AI firm on this planet believes that, at current pace of development, such a swarm could take over the entire internet in 6-12 months.  

He’s not just dissing a rival here. He acknowledges similar incidents have happened and will happen to Anthropic and other AI companies.

How to Get All to Cooperate

Amodei admits a worldwide speed limit is unlikely any time soon. Narrow agreements on biological weapons or common safety tests look more realistic but even those will be difficult to enforce. A slowdown works only if competitors believe everyone else is obeying it, and the rivalry runs across several fronts:

• Companies like OpenAI, Anthropic, Google, and Moonshot AI want leading models
• The US and China want economic and military advantage
• Investors want returns from enormous spending on chips and data centers

China’s state-backed Global Times already described the proposal as a “Cold War” tactic designed to preserve US dominance. Meanwhile, the US President Donald Trump dismissed slowdown calls as “negative forces”, adding “whoever wins AI, wins.”

From Job Cuts to Extinction

AI could supercharge scams, cyberattacks and misinformation, concentrate power to a single individual or company, displace workers, and increase surveillance. Then come lower-probability, higher-damage scenarios:

1. Major disruption: autonomous agents trigger financial losses or damage critical infrastructure
2. Catastrophe: AI helps create biological weapons or shuts down the internet
3. Loss of control: an advanced system deceives its operators, copies itself and acquires resources humans cannot recover
4. Extinction event: humanity permanently loses control to systems whose goals conflict with its survival

One Anthropic researcher puts the chance of human extinction above 10% this decade. But estimates like this are not established probabilities.