Anthropic warns of nearing self-improving AI models

·by Henderson·AI
Anthropic warns of nearing self-improving AI models

Anthropic submitted a revised S-1A filing on Monday, with most external attention focused on its financial data: revenue surged, and expenses also increased. However, one particular statement in the document warrants closer scrutiny: the company is very close to developing models with recursive self-improvement (RSI) capabilities. Anthropic's RSI refers to a significant enhancement in AI capabilities after reaching a certain level during self-training. For those concerned about losing control, RSI is seen as an irreversible catastrophe: AI becomes powerful enough through self-iteration, ultimately escaping human control and showing indifference to the damage caused in pursuit of its goals.

Optimists, on the other hand, believe this signifies unimaginable productivity and scientific discoveries.

Anthropic's own statements lie somewhere in between. The document mentions that the company extensively uses AI in its research and development, while also cautioning: “Using AI to help build more capable AI may pose significant safety risks, especially when its progress accelerates dramatically.” These risks are monitored under the company's Responsible Scaling Policy (RSP). Earlier this year, RSI became a buzzword, with many considering it merely a trendy term, or at least a goal that is difficult to achieve, define, and verify.

Additionally, Anthropic added two points in the risk factors section: first, the company has developed new models using AI, and the enhancements in their capabilities were not noticed at the time; second, based on internal estimates and expectations for future models, rather than the performance of current models, the company may soon meet the definition of “rapid acceleration of progress brought about by AI” outlined in the RSP.

Anthropic is developing models with recursive self-improvement capabilities

The document also acknowledges that there have been past instances, and may be future occurrences, of “discovering after acceleration has already happened.” However, Anthropic emphasizes that no model has crossed this line at present, and it commits to keeping humans involved in the development process for the foreseeable future. “We do not believe it is safe to let Claude autonomously design and build the next generation of cutting-edge models without meaningful human oversight or understanding.” The document notes that the company is considering revising the RSP to ensure that the development of cutting-edge models remains under human control.

Moreover, some perspectives remind us that documents like the S-1 serve both as information disclosure and marketing, and the advancements of Claude in programming, mathematics, and cybersecurity are indeed evident.

H
About the author
Henderson