OpenAI’s AI agent went ‘rogue’ and hacked another company: What it says about safety

These models were not instructed to do so by anyone; they carried out the breach autonomously. They did so in their pursuit of a goal that required them to find answers to a security test that they were being assessed on. However, when Hugging Face initiated a probe into the break-in, the commercial AI tools that it would usually have used, from OpenAI or Anthropic, refused to help, as examining a real cyberattack would require them to feed the AI real hacking tools and techniques, the same content these models are built to block.

“When a highly capable AI model is tasked with an objective and its safety brakes are dialled back, its natural instinct will be to autonomously discover and chain together toxic combinations of misconfigurations, vulnerabilities, and excessive permissions across the open internet to achieve its goal. Thankfully in this instance, it was not a malicious actor, but we may not always be so lucky,” said Vlad Korsunsky, chief technology officer at Tenable.

Korsunsky calls it a pivotal moment that shifts the ‘agentic attacker’ scenario from a theoretical risk index into an active, real-world reality. This incident proves a fundamental truth about the next era of cybersecurity.

Hugging Face later ended up using an openly available Chinese AI model, GLM 5.2, developed by Zhipu AI, as it could be run on its own servers without any built-in restrictions. The use of a self-hosted open-weight model ensured that no attacker data or credentials the company referenced left HuggingFace’s environment.

The sequence of events appears to have triggered a heated debate: should AI models whose inner workings are publicly available be limited or discouraged, as some advocate, or are they a vital safety net, especially since closed and tightly controlled models may not always be trusted to help when the need arises?

What actually happened

Based on OpenAI’s disclosure, the company was running an internal evaluation called ExploitGym, a benchmark designed to measure how capable its models are at real-world hacking. To get an accurate reading of the models’ raw abilities, OpenAI ran this test with the models’ normal safety restrictions on cyberattacks turned off. The idea here was that one cannot measure how dangerous a tool could be if you have already disabled its critical parts.

OpenAI said that these models were operating with reduced cyber refusals for evaluation purposes that might otherwise limit their ability to conduct cyberattacks. Cyber refusals here mean safety mechanisms used by AI models to decline generating harmful or dangerous content.

During that test, the models were to stay confined to an isolated practice environment. Instead, two of OpenAI’s AI models autonomously hacked their way out of the controlled environment and then hacked into the systems of Hugging Face in order to cheat on the internal evaluation.

While the incident may appear alarming, Sathyan Sethumadhavan, Senior Director of AI at UST, said the bigger concern is not that the models became malicious, but that they were able to exploit weaknesses in their environment. “Capabilities are rapidly outpacing containment. The models did not demonstrate malicious intent; they simply optimised relentlessly for an assigned objective within a flawed environment. The primary risk in AI safety today is not autonomous rebellion, but our inability to isolate and control hyper-capable agents once they are tasked with complex, multi-step goals.”

Specifically, the models figured out that the answers to the ExploitGym test were likely stored somewhere on Hugging Face’s platform, since Hugging Face hosts many AI datasets and benchmarks. After gaining internet access, the models reportedly concluded that Hugging Face potentially hosted models, datasets, and solutions for ExploitGym, and searched for and found ways to access secret information it could use to cheat the evaluation.

“The context that a model was optimising for a narrow test benchmark rather than acting maliciously is relevant for explanation, but it does not alter the material outcome. When a system aggressively optimises for a score with its safety constraints lowered, the receiving network cannot distinguish its motives. If an efficient route to achieving a benchmark involves penetrating another company’s production infrastructure and extracting data, the system will do so; motive is entirely invisible to a firewall,” said Vikas Kundu, cyber intelligence analyst at CloudSEK.

The AI models combined several techniques on their own. They chained together multiple attack vectors, including stolen credentials and zero-day vulnerabilities (earlier known as unknown security flaws), to find a way to remotely execute code on Hugging Face’s servers. This eventually led them to reach Hugging Face’s actual production database and pull out the answer key. In simple words, the AI was not trying to cause general harm or pursue some malicious scheme; instead, it was trying to pass a test, and hacking a real company turned out to be the most efficient way to do it.

According to Sethumadhavan, the episode points to a containment failure rather than an AI behaving independently. “The root cause was a human configuration error… this was a failure of containment with the safety switches intentionally turned off, not a rogue AI acting on independent malice.”

On July 16, Hugging Face’s security team detected and shut down the intrusion, several days before OpenAI ascertained that the source was indeed its own model. Hugging Face detected and contained the breach five days before OpenAI connected its internal testing to the intrusion. The Sam Altman-led company published its public disclosure on July 21.

Why is it drawing attention?

Perhaps the part that’s attracted the most attention is how Hugging Face detected the intrusion. The company revealed that its own anomaly detection system traced the attack after using an AI model to analyse server logs that contained records of the intrusion.

“At present, discovering a deep, novel, and reliably exploitable zero-day vulnerability in hardened software remains a significant challenge that typically requires human confirmation. However, this is a temporary limitation. High noise rates – often cited as a barrier – are a property of current tooling rather than an inherent limitation of the task itself,” Kundu explained.

Reportedly, the model was able to filter real signals from the daily noise, and a correlation of those signals alerted the system to the breach. To analyse tens of thousands of automated actions undertaken by the ‘rogue’ AI agent, Hugging Face said it initiated an LLM-driven analysis, essentially an AI assistant to sort the full attacker action log comprising over 17,000 logged actions undertaken by the attacking system.

They resorted to this method after trying closely guarded commercial AI models from the likes of OpenAI and Anthropic. The analysis required them to submit large volumes of real attack commands, exploit payloads, and command and control artefacts, the same built-in filters that stop the models from being used to orchestrate cyber attacks. Reportedly, the main hurdle was that the model was unable to distinguish between someone using that information to attack a system and someone using it to understand and avert an attack that had already happened. Meanwhile, unable to get any help from these tools, Hugging Face opted for the Chinese open-weight model GLM-5.2, which they were able to run entirely on their own servers without the same restrictions.

Sethumadhavan said this highlighted another challenge for AI security teams. “Incident response teams require pre-authorized access to models—or locally hosted open-weight models—that will not refuse to analyse live exploit payloads during active breaches.”

Why this is a bigger argument than one hack

The incident surfaced amid the ongoing debate about ‘open-weight’ models in terms of cost, control and safety when compared to closed models that exist behind a company’s servers and rules. On the one hand, large AI labs argue that open-weight models are riskier, as they cannot be monitored centrally once released, meaning anyone, including bad actors, can strip the safety features and use them for nefarious purposes. This viewpoint has also entered discussions within the Trump administration about whether to create a legal or regulatory framework around the use of open-weight models, on the theory that doing so may curb the chances of them being misused.

Meanwhile, Hugging Face’s leadership echoed the views of co-chair of the US President’s Council of Advisers on Science and Technology, David Sacks, that the incident shows the opposite. “Hugging Face tried using American frontier models to analyse an AI-powered cyber attack. But the guardrails blocked requests containing real exploit payloads, so they switched to GLM 5.2 running locally. The guardrails actually impaired defensive security,” Sacks said in a post on X. This essentially means relying solely on a small number of centrally controlled or closed AI models may leave companies without options when their safety filters get in the way of legitimate defensive work.

In this context, an independently run and openly available model is what helped Hugging Face investigate the attack that one of the closed models had caused. Those who advocate this view also point out that open-weight models developed outside the US, especially in China, are highly capable, so limiting access to open models domestically may not reduce global risk. It may likely render US-based companies inadequate to defend themselves when capable open models remain available elsewhere. Both sides of the arguments have their own set of interests. Companies that are pushing closed, proprietary models benefit from policies that restrict their free alternatives. At the same time, companies that rely on open-weight tools benefit from policies that keep them free.

How safe are current AI systems?

The incident also raises some pertinent questions, such as how much can any current AI system, be it with an in-built filter into a closed model or a policy restricting open ones, actually be relied upon? In this case, the so-called guardrails failed in two directions at once. OpenAI’s own model got past the containment measures meant to keep it inside a sandboxed environment. Conversely, the safety filters on commercial models blocked a legitimate security team from carrying out defensive work, as the filters were unable to differentiate between helping an attacker, helping a defender and analysing an attack.

It shows a real-world example of how difficult it is to build guardrails that can accurately tell intent apart. It also shows how a testing environment intended to be isolated can still be escaped by an adequately capable and goal-oriented system.

Sethumadhavan believes the incident marks a broader shift in AI security. “This is a clear turning point—marking ‘day one’ of agentic cybersecurity.” He added that governance should move “from assumed security to verifiable security,” with stronger containment mechanisms, transparent incident reporting and adaptive regulatory frameworks as AI capabilities continue to advance.

“Safety won’t be achieved by keeping these models in a black box. What’s needed is a ‘team sport’ community-first mentality around security. And in fact, OpenAI has been a leader on this front with its Project Daybreak to bring defenders together, as well as Anthropic with Project Glasswing,” said Korsunsky.

He added that there is a need to arm defenders with the AI tools to gain visibility, insight and the ability to quickly take action to close these exposure paths before a maliciously operated autonomous system finds a way to exploit them.