Anthropic told California Governor Gavin Newsom that officials should be able to ask a court to block the release of an AI model posing a “serious risk” to the public, in an October 15 letter Reuters reported on October 28. Politico first reported its details.
“When a review finds a model poses a serious risk, the government should be able to ask a court to block it,” the letter said. An Anthropic spokesperson confirmed the letter is authentic; the governor’s office did not immediately respond to Reuters.
That is a notable stance for a company that builds the very models it wants courts to be able to stop. Anthropic, the company behind Claude, filed publicly for a US initial public offering on October 26. Its letter asks for a judge in the middle, not a free hand for regulators.
Why Anthropic wants California courts to block risky AI models
Anthropic sent its recommendations after Newsom convened a panel of AI experts in September to develop proposals for AI safety requirements. The panel gave the state’s biggest AI labs a direct channel to shape what comes next, and Anthropic used it to draw a line between two very different kinds of intervention.
On one side is a court order aimed at a specific model. Politico reported that the state attorney general could seek one when a model presents a near-term risk of catastrophic harm, and a court could then suspend its deployment, internal use, or further development.
On the other side is a blanket shutdown switch. Anthropic opposes rules requiring developers to build “kill switches” able to shut down an entire model if a company loses control. Targeted steps would work better, it argued: disable a problem AI agent, or cut off one customer’s access, instead of pulling the whole system down.
It also backed regular public reporting and outside scrutiny: risk reports at least every six months from large developers, and independent evaluators working inside AI companies, paid through a state-run fund rather than directly by the companies they check.
Kill switches vs. targeted shutdowns
The kill switch debate is not abstract in Washington. House lawmakers proposed the “AI Kill Switch Act” in July, and a bipartisan group of senators debated a separate September proposal to block unsafe model releases, with companies able to challenge in federal court. Congress then left for its pre-election break without moving that proposal forward.
Anthropic’s letter fits into that gap. It supports shutdown capability, but wants the response matched to the problem: a single rogue agent is not the same as a model that threatens public safety, and shutting down everything could itself cause harm if hospitals, businesses, or researchers depend on the system.
Critics will read it differently: a targeted fix sounds reasonable until the problem turns out to be in the model itself, not in one deployment of it. Anthropic’s answer is the court order: when the risk is serious and near-term, a judge should be able to halt deployment, internal use, and development alike.
OpenAI did not go as far. Politico reported that OpenAI sent Newsom a similar letter last week supporting shutdown controls and embedded auditors, but stopped short of backing government power to block a high-risk model.
Why this matters for AI regulation
In June, the Trump administration ordered the company to immediately restrict foreign access to its cyber-capable Mythos 5 and Fable 5 models, citing national security concerns. The US Commerce Department later lifted that order. So Anthropic is also asking for a check on executive power: a judge, not an agency acting alone, would decide whether a model stays or goes.
That framing may be the smartest part of the letter. Courts already handle emergency orders in other industries, from product recalls to injunctions against dangerous drugs. Anthropic is effectively saying AI should get the same treatment: fast action when the danger is real, with evidence and a decision a company can appeal.
Still, plenty of questions remain. Who conducts the review that triggers a court filing, and what counts as a “serious risk”? How fast could a court act if a model is already spreading through developer tools? And would a California order reach models built elsewhere but used by millions of Californians? Anthropic’s letter sketches a framework, not a finished law.
What happens next
Newsom’s expert panel is the immediate next step. Its proposals will shape whatever bill reaches the California legislature, and Anthropic’s letter is now part of that record. Expect other labs to weigh in, and expect the kill switch question to keep splitting the industry: everyone agrees a dangerous system should be stoppable, but nobody agrees yet on who pulls the plug.
For users, that’s a distant but real change. If California adopts a court-block rule, a launch could face a legal pause before release, and a model already in use could be suspended in the state while a case is heard.
FAQ
What did Anthropic tell California Governor Gavin Newsom about risky AI models?
In an October 15 letter, Anthropic said government officials should be able to ask a court to block the release of an AI model that poses a serious risk to the public.
Does Anthropic support a kill switch for AI models?
No, not in the blanket sense. Anthropic opposes rules requiring a switch that shuts down an entire model, and instead backs targeted steps like disabling a specific agent or cutting off one customer’s access.
Would a court be able to stop a model that is already out?
Under Anthropic’s proposal, yes in extreme cases. Politico reported that a court could suspend a model’s deployment, internal use, or further development when it presents a near-term risk of catastrophic harm.
Has California acted on Anthropic’s proposal?
Not yet. Newsom convened an AI expert panel in September to develop safety proposals, and the governor’s office did not immediately respond to Reuters about Anthropic’s letter.
Sources: Reuters, Politico