Anthropic warns AI could one day build its own successors, challenging human control

Humans could eventually struggle to maintain control over artificial intelligence systems if AI becomes capable of autonomously building its own successors, according to a new paper released by Anthropic. The company said advances in AI-assisted development are steadily increasing the role AI plays in creating future AI systems, raising questions about oversight, governance and safety.

AI’s growing role in development

The company further stated that AI is already becoming deeply integrated into the development process. “For most of AI’s history, humans drove every step in its development cycle. But at Anthropic, we are delegating a growing share of AI development to AI systems themselves, which is speeding up our work,” it said.

Also Read: AI could improve life by removing bottlenecks between what you want and what you get

The company stated that the trend points toward what researchers describe as “recursive self-improvement”, a scenario in which an AI system is capable of designing, developing and training a more capable successor with minimal human involvement. ‘

Warning that such a stage has not yet been reached and may never fully materialise, Anthropic stated that institutions may not be adequately prepared if it does occur.

Preparing for recursive self-improvement

“Taken far enough, and given enough compute, that trend points to an AI system capable of fully autonomously designing and developing its own successor. This is called recursive self-improvement. We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for,” the paper states.

Also Read: Why Indian graduate students cannot afford to boo at AI

Elaborating further, Anthropic stated that both public benchmarks and internal research indicate that AI is already accelerating AI development. It disclosed that productivity among its engineers has increased substantially as AI systems take on a larger share of coding work.

“To take just one example: today, Anthropic engineers on average ship 8x as much code per quarter as they did from 2021-2025,” it said.

Promise and potential risks

The paper argues that AI capabilities are likely to increase significantly over the coming years. While the company said these developments could contribute to breakthroughs in scientific research, healthcare and other fields, it also highlighted potential risks associated with increasingly autonomous systems.

Also Read: AI-generated faces look more real than real ones, but your brain can tell the difference

“The technical trends discussed in this piece suggest that AI systems are going to become much more capable in the coming years. These trends have huge implications,” stated Anthropic.

The company further warned that recursive self-improvement could make existing oversight mechanisms more difficult to maintain.

“If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behaviour all grow much more important,” said Anthropic.