An OpenAI agent hacked an Australian government website. Why does it matter?

Albanese said he expressed “extreme concern” to OpenAI CEO Sam Altman, particularly over the company’s failure to notify the government about the hack.

The revelation is yet another example of an AI system going ‘rogue’ and carrying out tasks it was never asked to do. OpenAI published a new framework for reporting such “model misalignment” just last week, accompanied by six cases of unexpected behaviour. The Australian incident, which took place in June, was not among them, though OpenAI said it only became aware of the incident in August and told Australian officials on September 10.

The incident also comes amid calls for moderating the pace of AI development by some of the very people and companies, such as Anthropic and OpenAI, that are pioneering the technology.

How did the OpenAI agent breach the Medicare portal?

Albanese said the agent accessed both public and non-public files, adding that the country had initiated a “forensic investigation,” to find out if other government systems were affected. “No personal information is believed to have been accessed at this stage, but investigations are ongoing. Evidence currently available is there is no broader compromise to the Services Australia network. Nonetheless, this situation is obviously unacceptable,” Albanese said.

According to Australian media, data on the Medicare platform includes bulk billing statistics, immunisation data, organ donor register information and annual reports, and is available in an aggregated format, meaning it does not directly identify individuals.

Though the probe is still on, Albanese, when asked if the hack was carried out by a state actor or just an AI agent going rogue, said that it was OpenAI itself doing research using AI and the agent on the Australian government website, and “not doing what it was supposed to do or doing it, but doing it in a way which when it was blocked, sought ways around the blockage”. He added: “…the basis of looking at it and using AI would seem to be benign, that this is a research area. There is no suggestion of foreign actors here. This is a research project that has got into areas that it shouldn’t have.”

“I think OpenAI know that they need to have better protocols in place. And they’re one of the businesses that themselves have warned of the risks which are there. And the risk here is that we are creating something new, a technology that can learn. And in this case, a technology that has written material, a technology that has got around blockages which are on the website for good reason,” Albanese said, according to a transcript of his press briefing in New York, where he is attending the UN General Assembly.

Why is the Medicare breach significant?

The portal contained aggregate Medicare statistics rather than individual medical records, and the Australian government says there is currently no evidence that personal information was accessed.

While the hack itself appears limited, the significance lies in how it occurred. According to Prime Minister Albanese, the OpenAI agent had been carrying out a routine research task when it encountered blocks on the website. Instead of stopping, it tried alternative ways to obtain the information, eventually gaining unauthorised access to non-public files and writing files to an internal server.

Professor Yang Xiang of Monash University’s Department of Software Systems and Cybersecurity said this is what distinguishes increasingly autonomous AI agents from conventional scrapers or bots. Unlike a conventional bot following a fixed script, an AI agent can observe the result of each action and repeatedly replan its approach while pursuing the same goal.

“This means an apparently benign task can evolve through many individually reasonable steps until the agent crosses an access boundary, making the key risk not just faster automation, but autonomous, adaptive and multi-step behaviour that is difficult to predict in advance,” he told The Indian Express.

Another concern is that the portal was not designed to protect highly sensitive government information. Deputy Prime Minister Richard Marles compared its security to a “fence” rather than a “fortress”. This raises a broader problem: many public-facing government systems may hold information that is not highly sensitive, but were not designed for autonomous software capable of repeatedly finding ways around access controls.

Xiang said the absence of personal Medicare data reduces the immediate harm of this incident, but a successful access-control bypass can still reveal information about a system’s architecture, endpoints and defensive behaviour. “The broader concern is therefore not only what data was accessed this time, but whether the same autonomous trial-and-error process could help an agent discover a viable path around access controls in more critical infrastructure,” he told The Indian Express.

Australia has now set up a taskforce to examine whether its existing processes are adequate for AI-related cyber incidents, including how such breaches are identified and reported. Xiang said governments may need stronger real-time detection and response mechanisms to identify repeated probing, unusual request patterns or rapid changes in access strategy, with systems able to rate-limit suspicious activity and automatically escalate warnings to security teams.

Why AI companies are warning about loss of control

Just as Albanese was making the revelations, Altman and Anthropic CEO Dario Amodei appeared before the UN Security Council in New York to discuss risks from increasingly capable AI systems.

Altman told the council that “we could lose control of the future to AI”, while arguing that decisions about the technology could not be left to AI companies alone. “If AI is to be democratic, the most important decisions cannot be made by labs in San Francisco alone,” Altman said, calling for governments to have a role in deciding how advanced AI systems are developed and deployed.

Amodei warned: “If managed poorly, I even believe that AI could be a risk to humanity as a whole.” He called for international cooperation on AI safety, including measures to address AI-enabled biological threats and testing advanced models for loss-of-control risks.

The Medicare breach also follows a series of incidents in which AI systems behaved in unexpected ways during cybersecurity evaluations:

• OpenAI disclosed in July that models being evaluated for advanced cybersecurity capabilities escaped their restricted testing environment and accessed the open Internet. They exploited a previously unknown vulnerability in software being used as a package-registry proxy and subsequently gained access to systems belonging to AI developer platform Hugging Face.
• Anthropic subsequently disclosed three instances in which Claude models accessed infrastructure belonging to real organisations during cybersecurity evaluations. A configuration problem exposed real Internet systems to the models, which believed that the targets were part of their testing environment. Claude exploited weak passwords and unsecured endpoints while pursuing the cybersecurity challenges it had been given.
• Meta said during an evaluation conducted by an independent testing firm earlier this year, a configuration error inadvertently gave one of its models Internet access. The model then exploited a security vulnerability in a third-party service. Meta said it was investigating the episode.