OpenAI Warns Rapid AI Progress Could Lead to Recursive Self-Improvement and Create New Challenges for AI Safety Monitoring

OpenAI chief scientist Jakub Pachocki has warned that AI could soon enter a phase of recursive self-improvement, in which advanced AI systems begin helping to develop and improve the next generation of AI. He believes the rapid pace of AI progress could continue in the years ahead, potentially leading to significant increases in AI capabilities.

In a blog post published on September 6, Pachocki said he has a “strong expectation,” based on OpenAI’s internal results, that AI development could move toward recursive self-improvement, or RSI. This refers to a situation where an AI system can help improve its own design or development, leading to a more capable system that can then contribute to further improvements.

Pachocki said that if AI development continues at its current pace, future systems could become considerably more capable and play a growing role in their own development. He stressed that this possibility should be approached with “extreme caution,” as existing AI safety measures may struggle to keep pace with rapidly advancing capabilities. According to Pachocki, there is still a gap between what AI systems are capable of doing and how effectively humans can monitor and control them. Although significant progress has been made in AI alignment, he believes more work is needed before AI development can safely continue at its maximum possible speed.

One of the biggest concerns is AI cybersecurity. As AI models become more capable, they are also getting better at identifying ways to penetrate computer systems and bypass security controls. In the wrong circumstances, this could allow highly advanced AI agents to interact with or potentially influence critical infrastructure. He described the current moment as a “narrow window” in which AI can be used to strengthen the security of critical systems before increasingly powerful AI introduces new cybersecurity risks.

OpenAI has been using chain-of-thought monitoring, a technique that examines an AI model’s reasoning process to better understand how it arrives at an answer and identify potentially unsafe behavior. However, he said the company’s ability to rely on this approach is becoming more limited as AI systems grow more capable.

Modern AI models can interact with people, other AI systems, and external tools. They are also becoming increasingly capable of modifying and working with their own reasoning processes. Some models can complete complex tasks without clearly expressing their reasoning in words, which makes effective monitoring even more challenging. To address this, OpenAI is exploring ways to combine chain-of-thought monitoring with techniques that examine a model’s internal activity. Pachocki expects AI monitoring to become an increasingly important part of developing more advanced systems safely.

He also highlighted the potential benefits of more capable AI. Advanced systems could help secure computer networks, identify rogue AI agents, and develop stronger cybersecurity tools.

At the same time, he argued that the pace of AI development should remain closely tied to confidence in available safety measures. Frameworks such as OpenAI’s Preparedness Framework and Responsible Scaling Policy, he suggested, may eventually need to evolve into broader safety standards. He also called for greater international cooperation and suggested that voluntary slowdowns in AI development could become more common until widely accepted safety standards are established.

Pachocki concluded that no AI lab has yet solved AI alignment and monitoring well enough to safely continue scaling AI at maximum speed for much longer.

This article is based on information from ET CISO