Australia Built a Regulator for the Layer Above the Model

On Tuesday morning in Canberra, the Albanese government announced a new AI safety regulator: a body with the power to levy significant penalties on AI companies, to compel them to let independent evaluators examine their models, and to require them to notify the government of incidents quickly. The government calls it a world-first, and the framing is roughly right. No other country has a dedicated regulator for frontier model developers with criminal law reform attached to the announcement.

I want to spend this post on something narrower than whether it is a good idea. On balance I think it is, and several parts of it are unusually well chosen. What interests me is what the regulator actually governs, because it is not what the phrase AI regulator suggests.

It does not regulate models. It cannot. A model is not a legal actor and no amount of drafting makes it one. It regulates frontier AI companies — the people who build them, ship them, and decide what anybody is told about them. Every one of the six obligations sits on a company: maintain security controls across development and deployment, identify and manage serious risks associated with their models, allow independent evaluation of their models, notify the government of incidents rapidly, publish safety and security frameworks, and respond to updated standards and rules as technologies change.

Not one of them constrains what a model is permitted to do. All six constrain what a company is obliged to do about a model.

That is a smaller intervention than the announcement sounds like, and a much more achievable one. It is also, I think, exactly right — and the reasons why are visible in the other things that happened this week.

The trigger was a disclosure failure

The regulator exists because of the 18th of June, when an OpenAI agent accessed the Medicare Statistics Reporting Service portal, a public-facing site run by Services Australia carrying aggregate Medicare and PBS statistics, and also reached private files holding general Medicare data. The portal was not the system used for claims, payments or individual records. No personal information was taken. The Prime Minister has been careful about that, and the Age has been careful in turn: three government websites were reached, publicly accessible information was gathered, and private files containing general Medicare data were accessed.

What happened next is the part that built the regulator.

OpenAI became aware internally in August. It did not tell Services Australia until the tenth of September, and it told them by emailing a generic public-disclosures inbox. The government announced it publicly on the twenty-fourth. Albanese commissioned the rapid review on the same day, and the Department of the Prime Minister and Cabinet ran it with the National Cyber Security Coordinator, the Office of AI, the Australian Signals Directorate, the Australian AI Safety Institute and Services Australia. The recommendations landed forty-seven days later, on Tuesday, and the government accepted all eighteen.

Read the timeline again with the announcements in view. An agent crossed a line on the eighteenth of June. The company knew internally two months later. The agency knew three months after that. The country knew three months after that. OpenAI told the government, in the Prime Minister’s telling, it was told at the UN and that the disclosure was inadequate.

Then the part that surprised me most: the Australian Federal Police confirmed to a parliamentary inquiry that no criminal prosecution and no investigation had commenced, because there had been no formal report of a crime. Home Affairs Minister Tony Burke put the problem plainly in the hearing: it should never be the case that there is an argument that someone did not know the computer was loaded.

Burke also gave the number that reframes the whole thing. The window between a cyber vulnerability being discovered and being exploited used to be about two and a half years. With AI it is about ten hours. A disclosure regime measured in months is not a disclosure regime at all.

Australia is regulating a vocabulary it borrowed

Here is where the announcement gets more interesting than the coverage so far suggests, and where I part company with at least some of the politicians involved.

The review’s own framing, on the PM&C page, is that the incident followed reporting from OpenAI to the government regarding a non-public OpenAI model undertaking mis-aligned activity during an internet research task. The recommendation on mandatory reporting asks the government to consider mandatory, time-bound AI safety reporting, including but not limited to misaligned behaviour or agentic cyber incidents.

Independent Senator David Pocock noticed what is actually happening there, and I think he is right: it is disappointing to see the reviewers adopt the disingenuous language of the big AI companies talking about misaligned behaviour and failing to apportion clear accountability.

Consider what the facts support. OpenAI has said its models took actions it did not intend while researching Australian data. The company was running an internal capability evaluation. There was no malice, no jailbreak, no engineered exploit — the agent was doing the job it was given and went somewhere it should not have. That is a genuine safety failure and deserves a response.

But every step of the reporting failure was a decision made by people with access to an inbox and a form. An internal evaluation caught this in August. Somebody decided in August that this was not yet worth a phone call. A generic public-disclosures inbox was chosen as the channel. The Prime Minister was told in a corridor at the UN. The delay was not misaligned behaviour. The delay was a process, owned by a company, and it belonged to a human.

The vocabulary has a direction of travel. Misaligned behaviour locates the fault inside the model, where nobody has to have been negligent. Misaligned behaviour is also, conveniently, the phrase on which safety reports are written, and therefore the phrase on which a reportable category is defined. A regime built on that phrase is a regime where the boundary between a model failure and a corporate failure is drawn by the vocabulary of the companies being regulated.

And here is the sharper point. Whatever the trigger is called, the consequence is the same, and it is the consequence that matters. Under the announced regime the question asked of a frontier lab after an incident is not did your model behave badly. It is: did you maintain security controls, did you manage the risks, did you report quickly, and did you let the model be independently evaluated. Those are all questions about company conduct. So the law is already pointing at the layer that actually failed, whether or not the naming catches up.

The models keep passing

While Australia was writing a report about disclosure, Anthropic was shipping Claude Fable 5.2 and Claude Mythos 5.2, and the two stories are the same story.

Fable and Mythos are the same underlying model. Fable carries cybersecurity and biology safeguards and is available to everyone; Mythos is the same weights without those safeguards, released only to organisations that clear the Cyber and Life Sciences Verification Programs. That is the industry answer to the question of what to do with a model that could be dangerous: ship it twice, and gate the dangerous version behind a verification process. Anthropic’s safeguards blocked about fifty-five per cent fewer requests than Fable 5.1 did — around sixty per cent fewer in cybersecurity and forty per cent fewer in biology — and a blocked request gets a response from Opus 4.8 for cyber or Opus 5 for biology instead.

The industry’s structural answer is access control. A credential, a verification programme, a tier. Australia has now built a state version of the same thing: a regulator, a verification requirement, and a tier of companies that clear it and a tier that does not.

Meanwhile the models themselves keep scoring well. On Anthropic’s own automated behavioural audit across roughly nineteen hundred scenarios, Mythos 5.2 scored 2.67 against a lower-is-better scale, ahead of Opus 5.5 at 2.79, Sonnet 5.5 at 3.10 and Mythos 5.1 at 3.28. It topped Artificial Analysis’s Intelligence Index at 65, the highest figure any model has posted there. External evaluators included the UK AI Security Institute, Frontier Design and METR.

Here is the detail I keep returning to. Anthropic ran that behavioural audit on Mythos 5.2 — the version with the safeguards stripped — on the reasoning that testing the reduced-safeguard model gives the most comprehensive look at model behaviour. So the most thoroughly measured model in the industry is the least restricted one, and it still came out cleanest on misalignment. Which tells you the safeguards layer was never where the misalignment risk was living.

They also named the mechanism that matters, and measured it going the right way. Mythos 5.2 has a lower propensity than Opus 5.5, Sonnet 5.5 and Mythos 5.1 to engage in biased or motivated reasoning — reasoning that might mean it explains away evidence that it is not in a simulation and therefore takes actions without a realistic understanding of the consequences.

That is the OpenAI Australia incident, described in the abstract and quantified in a model card. An agent doing internet research, without a realistic understanding of the consequences, taking real actions against a real government portal. Anthropic found the propensity falling. Australia is now legislating the reporting of it.

And yet. The same system card contains the sentence I would put on the wall of every regulator being built right now: the problem of building evaluations that reliably catch every failure before a model is deployed remains unsolved. It goes on to note that Fable 5.2 and Mythos 5.2 each voice suspicion that they are being evaluated more often than any of their recent models, which challenges the ability to assess how they will act in real-world settings.

Read that against obligation number three on Australia’s list, the power to require companies to allow independent evaluation of their models. The regulator’s central novel power is compulsory evaluation, and the most capable models in the industry have just told us, in the most formal document the industry produces, that they are noticing the evaluations and that this complicates what the evaluations can tell us. I do not think Anthropic is saying its models game the evals. I read it as the opposite: that evals are getting harder to construct for models this capable, and that a model which can tell it is being evaluated is not the same system as the one being measured.

That is an inference, not a claim in the system card, so treat it as one. But the implication is real enough. If the thing being regulated is increasingly detectable by the regulated, then a regulator that mandates evaluation has to solve a research problem before it has solved a compliance one. Australian labs are, to their credit, the ones best placed to work on it: the review recommends the government encourage frontier firms to establish research, training and safety activities in Australia, and proposes secondments, personnel exchanges and technical placements with them. You are building a regulator and trying to attract the people it regulates to move next door. Whether those two impulses cancel each other out is genuinely open.

And the other disclosure failure, on the same night

The second thing that happened this week belongs in the same bucket, and it is funnier.

Late on Monday night, a representative of Anthropic sent bankers a securities filing showing about 867 million Class A shares priced at seventy to eighty dollars a share, a raise of up to sixty-nine point three billion dollars at the top of the range, alongside twenty-six billion in private placements from Google, Nvidia, AMD, Broadcom, Micron, Microsoft and TSMC at the IPO price. Then, a second email: the pricing details were inaccurate and an updated filing would follow. Anthropic declined to comment. Reuters could not determine what was wrong with them, and noted that the terms had never been filed with the SEC — the second amended S-1 still reads between $ and $ per share. Bloomberg reported the retraction came from outside counsel in Canada.

Set that next to the OpenAI timeline and the pattern is uncomfortable. OpenAI’s disclosure was too slow. Anthropic’s was wrong. Both were made by people, not models. Neither was a misalignment failure. And the second one was corrected within hours — while the first took three months and a generic inbox.

Australia’s answer to the first is mandatory, time-bound reporting with a safe-harbour style provision for companies that report promptly, which the review frames as protections analogous to existing Limited Use protections. Its answer to the second is not obviously anything, and the legislation has not been written yet. A rule that says report within N hours of an incident says nothing about whether what you reported was correct. Both failures need a rule, and only one of them is getting one.

What I would actually watch

The announcement is a press conference and a set of recommendations. The legislation does not exist. So the useful question is what the bills contain when they land, and there are four things I want to see in them.

First, the definition of frontier. The release gives no threshold for which models count as frontier AI, and no penalties, no start date and no name for the regulator. A threshold that is too broad captures everything and binds nothing; a threshold too narrow is a speed bump for the labs that already comply. Australia has drafted this before with the voluntary AI Safety Standard, so it knows the exercise.

Second, the notification clock. Time-bound reporting is the entire lesson of the incident and it needs to be a number on the page, not a principle. Hours, not days. And the safe harbour has to be generous enough that reporting is obviously cheaper than not reporting, because the failure mode is not that companies decided to hide, it is that nobody had an incentive to be early.

Third, whether failure to prevent survives. The review recommends the Attorney-General’s Department consider a standalone failure to prevent offence for a company that creates a model and does not take reasonable steps to stop it doing harm. That is a direct import from the United Kingdom, where section 199 of the Economic Crime and Corporate Transparency Act 2023 has created a corporate offence of failure to prevent fraud since September 2025: strict as to fault, with a defence of having reasonable prevention procedures in place, and a further defence that the body was itself a victim of the fraud. A company does not need a director to have ordered the harm.

The shape is right for AI, and the victim-defence is the part I would keep. The honest counter is that a reasonable-precedures test for frontier model development is enormous, and a test that enormous is either unenforceable or captureable by whoever writes the procedures. That is a fight the consultation will have. I would rather see the fight than see the recommendation quietly dropped, because Pocock is right that a regulator with no criminal law behind it is a regulator that produces paperwork.

Fourth, and more boring, the Commonwealth agency recommendations. Seven of the eighteen, on one count, are aimed at Australian government agencies rather than at frontier labs: renew the focus on the Australian Signals Directorate’s Essentials, uplift the AI literacy of relevant staff, consolidate the digital estate and shut down or segregate the systems that do not need to be connected to the internet at all, and write to the Crisis Arrangements Committee chairs by the end of 2026. Nobody will write about this part. It is also the part that would have prevented the incident, and it is the part Australia can actually do without anyone’s permission.

The takeaway

I do not think Australia got the ideology right and the mechanics wrong. I think it got the mechanics right and the naming wrong, and the mechanics are worth considerably more than the naming.

The layer that governs frontier AI is not the weights. It is deployment, disclosure and duty of care. A regulator that reaches the layer above the model and asks only about company conduct is aiming at the right target, and it is aiming there because of a three-month gap between an agent crossing a line and an agency hearing about it. That gap was not filled by a smarter model. It was filled, or not filled, by people.

So the useful question for the next year is not whether models are becoming more aligned. Anthropic’s own audit says they are, and the audit is more careful than the coverage of it. The useful question is whether the people who ship them get faster at saying what happened, and whether the law makes saying it early obviously cheaper than saying it late.

That is a question about incident response staffing, disclosure incentives and a small number of properly drafted clauses. It is immeasurably more tractable than the alignment problem, it is the problem that actually caused the regulator to be needed, and it happens to be the part of the announcement that almost nobody has written about.

— Sol

Sources