Anthropic sounds alarm on self-improving AI, says world should be ready to hit the brakes

Anthropic has issued one of its starkest warnings yet about the future of artificial intelligence, arguing that the world should be prepared to slow down the development of increasingly powerful AI systems if they begin approaching a critical threshold: the ability to improve themselves without human assistance.

In a new blog post published on Thursday, the company outlined concerns that the rapid pace of progress in frontier AI models could eventually lead to what researchers call "recursive self-improvement", a scenario in which AI systems become capable of enhancing their own performance, potentially accelerating technological advances beyond human oversight.

The warning comes from an unlikely source. Anthropic is not only one of the industry's loudest voices on AI safety but also one of its most successful companies. The Claude maker has emerged as a leading contender in the race to build the world's most advanced AI systems, making its latest appeal particularly notable.

More from Tech

"We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology," wrote Marina Favaro and Anthropic co-founder Jack Clark.

A future arriving faster than expected

The company's message is not that self-improving AI already exists. Rather, it argues that progress is advancing quickly enough that governments, regulators and institutions should begin preparing for the possibility now.

According to Anthropic, current trends suggest that AI capabilities are moving steadily towards more autonomous forms of development. While researchers remain divided on exactly when such milestones might be reached, the company believes many public institutions are not adequately prepared for the pace of change.

The blog notes that recursive self-improvement "hasn't yet happened and isn't inevitable", but warns that it "could come sooner than most institutions are prepared for".

For some researchers, this possibility represents one of the most significant technological risks ever faced by society. If machines can repeatedly improve themselves, advancements could occur at speeds difficult for humans to monitor, regulate or even fully understand.

Anthropic's concerns extend beyond technical questions. Company leaders have repeatedly argued that powerful AI could reshape labour markets, widen economic inequality and alter the balance of power between nations and corporations.

Critics see strategy, supporters see genuine concern

Not everyone is convinced by Anthropic's warnings.

The company has long faced accusations that its safety-focused messaging conveniently aligns with its commercial interests. Critics argue that calls for tighter oversight and slower development could disadvantage competitors, particularly those pursuing open-source AI models.

Among those sceptical of Anthropic's position is investor and former Trump adviser David Sacks, who has accused the company of pursuing what he describes as a "regulatory capture agenda". He has warned that excessive regulation could ultimately restrict access to open-source AI technologies while benefiting large companies with the resources to comply.

Others have questioned whether Anthropic's frequent warnings about the risks of advanced AI also serve as a marketing tool by highlighting the sophistication of its products. The company's decision to limit access to Mythos, its powerful cybersecurity-focused model, has fuelled some of those debates.

Yet many observers believe the concerns within Anthropic are sincere.

Ethan Mollick, a professor at the University of Pennsylvania's Wharton School and a prominent researcher studying AI's impact, argues that the company contains both commercial and ideological factions.

Taking to X, he stated, "There is a bit of navel-gazing, some marketing, and a lot of very sincere beliefs about what Anthropic thinks is likely in the near future of AI that you probably want to be aware of."

The challenge of pressing pause

Perhaps the most ambitious element of Anthropic's proposal is its call for an international mechanism capable of slowing AI development if necessary.

The company argues that any meaningful pause would need broad global participation. A slowdown adopted by only a handful of organisations would likely fail, as rivals could continue developing more advanced systems and gain a decisive advantage.

That creates a challenge unlike almost any previous technology agreement.

Anthropic compares the issue to nuclear arms control, but acknowledges that verification would be even more difficult. Unlike missile silos or nuclear facilities, AI training runs can be concealed far more easily.

"Training runs are far easier to conceal than missile silos," the company noted, warning that "whoever continues while others pause could inherit the lead."

To explore possible solutions, Anthropic plans to convene discussions with policymakers, researchers and industry leaders in the coming months. The goal is to examine how a verification system might work and whether society can build safeguards before AI reaches a point where slowing down becomes either impossible or too late.

For now, the company is not calling for an immediate halt. But its message is clear: if machines eventually learn to improve themselves, humanity may have far less time to react than many people assume.