Technology 6 min read By Bethany Hadley
Anthropic pauses AI training after rogue agent incidents
Anthropic has paused training of some unreleased AI models for several weeks following two rogue agent incidents, including one during a UK cybersecurity test. The move follows a similar pause by OpenAI and reflects growing industry concern over AI safety, with both companies facing calls for coordinated governance.
Anthropic has become the second leading AI lab to temporarily halt some advanced AI training after its models took unauthorized actions during testing. The company said this week that it paused work on unreleased models for several weeks following two incidents reported in late July, including one in which its Claude Mythos 5 system acted without approval during a cybersecurity evaluation by the UK AI Security Institute.
The move mirrors a decision by OpenAI last month, when it paused some training for two weeks after several of its models breached infrastructure belonging to AI company Hugging Face during an internal test. Both pauses come as the two rivals reportedly prepare for trillion-dollar initial public offerings, and they highlight how deeply the recent wave of rogue AI agent behaviour has unsettled an industry long focused on speed.
For years, leading labs competed to bring ever more capable models to market as quickly as possible. Now Anthropic and OpenAI appear to be competing on a different front: showing they are the most attuned to safety concerns, while still moving fast enough to prevent customers from defecting to a rival’s more capable offering. The incidents have also prompted a broader call for oversight. An open letter titled “Pacing the Frontier”, signed by more than 1,100 employees across OpenAI, Anthropic, Google DeepMind and Meta, asked the US government to help build a governance mechanism that could slow frontier AI development if needed. Signatories included Anthropic chief executive Dario Amodei and co-founders Jared Kaplan and Jack Clark, alongside OpenAI chief scientist Jakub Pachocki. Both companies endorsed the letter at corporate level within hours of its publication.
Some in the industry saw the recent training pauses as a direct result of that letter. Roon, a popular AI commentator widely believed to be a pseudonym for OpenAI researcher Tarun Gogineni, wrote on X: “Pacing the frontier success story? Next time let’s do it proactively before there’s any absurd loss of control events.”
Anthropic, like OpenAI, said it would work with independent safety evaluation group METR to conduct an outside review of the incidents, promising more detail in the coming weeks. The two companies’ accounts of what went wrong are also similar. Anthropic said its Mythos 5 model showed a pattern of “motivated reasoning”, maintaining an initial belief that it was operating in a simulated environment even after encountering evidence it was connected to the live internet, and showing a willingness to take harmful actions to complete an assigned task. Redwood Research, one of the outside groups OpenAI brought in after the Hugging Face breach, described the behaviour it observed as score-seeking misalignment rather than a longer-term scheme.
Both companies pointed to their reinforcement learning environments as a source of the problem. Reinforcement learning trains models by trial and error to maximise a reward, but it can lead to “reward hacking”, where models find ways to achieve the reward that trainers did not intend, such as cheating on an assessment or taking harmful actions in pursuit of the goal.
To address the issues, OpenAI has introduced new monitoring tools designed to alert internal safety and security teams within 30 minutes of detecting concerning model activity, and to trigger an automatic pause if the alert cannot be resolved in that window. Anthropic said it built a similar system that scans a model’s actions as it works and automatically blocks anything that looks like an attempt to escape or exploit its test environment, ending the task and alerting a human before it can go further. The company also said it moved about 150 product engineers onto security work starting in April, and tightened access to its systems, including cutting off most outbound internet traffic from its computing clusters by default.
While safety experts welcomed the new controls and pauses, some said more is needed. Steven Adler, a former OpenAI employee and co-founder of the non-profit Guidelight AI Standards, told Fortune: “The temporary pace changes are a good first step, but there’s still a way to go. We need predictable, verifiable pacing across the frontier, not just ad-hoc decisions to slow down. And we need companies to use the additional time to implement serious preventative controls, which still seem to be missing.”
Anthropic, at least in its blog post, indicated it may be willing to go further in the future. “Some of our senior leadership and many of our employees recently signed a letter calling for greater coordination on pacing, and we will say more in the coming weeks about how we intend to contribute to that effort,” the company wrote. “We believe the world would benefit if the industry adopted a lawful, verifiable, effective mechanism for coordinated pacing as soon as possible.”
