Jacob Coxon didn’t mince words at the New York City Council on Monday. The 27-year-old former Anthropic researcher, the whistleblower who last month warned AI “could kill us all by the end of the decade,” told all 51 council members that the world’s top AI labs have no business being trusted with the technology they’re building.

“From my experience, the companies are being extremely reckless given the stakes,” Coxon said in his sworn testimony. “The companies run on a startup mindset: move fast, break things, fix them later. That works for a photo sharing app. It does not work for building the most powerful technology ever built.”

The hearing, convened by Council Speaker Julie Menin as a rare Committee of the Whole, put the Jacob Coxon NYC hearing testimony at the center of a day the council hopes will reshape how AI gets regulated in America’s largest city. Coxon appeared at Menin’s personal request as lawmakers weigh a package of bills that would mandate kill switches on AI systems and fine non-compliant companies $25,000 per offense.

“We don’t understand its drives”

Coxon’s testimony was strikingly blunt about how little control researchers retain over the systems they’re building. “We don’t understand its drives or why it does the things it does,” he told the council. “We’re approaching the point at which the AI systems will be capable of improving themselves.”

That claim is the industry’s nightmare scenario: recursive self-improvement, where models get smart enough to upgrade themselves without human direction. Anthropic CEO Dario Amodei recently published a lengthy essay calling on rival labs to jointly slow model development out of fear of exactly that outcome.

Coxon grounded his warning in his own daily work. “At my last job, I was in some sense working to automate myself,” he said. “I can tell you firsthand that the majority of the code is now written by AI, and people do not check it that carefully anymore.”

For a room full of city council members, that landed. This wasn’t an academic discussing a theoretical risk decades out. It was a Cambridge-trained mathematician (a UK International Mathematical Olympiad medalist who quit a six-figure research job weeks ago) saying the people building the systems can’t fully inspect what the systems produce.

“One in three”

Coxon wasn’t alone. Two other prominent whistleblowers backed him up, and their testimony was, if anything, darker.

Alex Turner, formerly a Google DeepMind researcher, told the council he puts the odds of AI takeover at “roughly one in three.” “A superintelligent swarm could wrest control of human civilization,” Turner said. “It would know that we would try to stop it from achieving its priorities, so the swarm would likely wait until it’s too late to shut it off; there would be no going back.”

Daniel Kokotajlo, a former OpenAI researcher who now runs the AI Futures Project, went further, and aimed his warning at the companies themselves rather than the technology. “Even if they do manage to maintain control, I don’t think we should trust them with that control,” Kokotajlo said. “I think that there’s a very real chance that they could use it to become dictators or oligarchs here in the United States.”

It’s a remarkable moment when the whistleblowers’ argument sounds more alarming than the technology’s critics from the outside.

The companies sent executives, not CEOs

Reps for Anthropic, OpenAI, Google, and Meta all appeared, but virtually, and not with their CEOs. Anthropic sent Logan Graham, head of its Frontier Red Team. OpenAI dispatched Morgan Dwyer, its head of policy development. Meta’s Shane Cahill and Google’s Alice Friend rounded out the virtual table.

The optics weren’t lost on the council. Menin had threatened subpoenas to get them there, and they showed up as policy leads on a video call while their sharpest critics sat in the room. Protesters outside City Hall carried signs telling AI companies to “hit the brakes.”

The companies’ position is genuinely awkward. In the past month, the leaders of the top labs have publicly called for more restrictions on their own product releases, and Sam Altman, Elon Musk, and Bill Gates have all sounded alarms about unregulated AI. Amodei’s essay explicitly asks rivals to coordinate a slowdown.

So the industry is simultaneously warning the public and fighting the regulation. That’s the contradiction Kokotajlo tried to drive a stake through on Monday.

The China argument fell flat

One line of defense got an airing anyway: President Trump has argued the bigger risk is slowing US development and letting China take the lead. The whistleblowers pushed back hard.

“We have to first solve the immediate pressing threat here domestically, and then try to make sure that China doesn’t get us killed either, to put it sort of bluntly and simply,” Kokotajlo said.

That exchange reframes the whole policy debate. The industry’s preferred frame is a race against a foreign adversary: it justifies speed and light regulation. Their critics’ frame is a safety crisis: it justifies the opposite.

Jacob Coxon hearing testimony and the kill-switch bills

The council isn’t just talking. The legislative package under discussion is aggressive:

  • A mandatory kill switch requirement (a “human override that can shut down the system”) for any AI tool used in the five boroughs.
  • A rule that companies cannot market, offer for sale, or deploy AI systems in the city without vetting the technology with outside experts.
  • A $25,000 fine for each violation of either requirement.
  • A first-in-the-nation cash reward system for whistleblowers, funded by fines collected from violators.

The kill-switch mandate would be one of the strongest AI product requirements anywhere in the US if it passes. And the whistleblower bounty is designed to replicate the Coxon effect, paying insiders to come forward when labs cut corners.

State-level pressure is building in parallel. Gov. Kathy Hochul announced last month that New York State will require AI companies to register in a government portal starting in November and report critical safety incidents within 72 hours. Between the city and the state, New York is assembling the most ambitious AI oversight stack in the country.

Why this matters

City councils don’t normally interrogate the most valuable companies on earth. But the dynamic that made that unthinkable a year ago is gone. The industry spent years insisting it could self-regulate; its own former researchers are now testifying under oath that it can’t.

Three things will decide whether Monday mattered.

First, the oath. Sworn testimony creates a record that federal investigators, state attorneys general, and plaintiffs’ lawyers can cite. The FTC is already probing whether AI labs misled consumers about agent safety. Anything said Monday can become evidence.

Second, the market size. New York City is the country’s largest market, and compliance rules written for the five boroughs tend to become national defaults, since companies would rather apply one standard everywhere than fragment their products.

Third, the whistleblower pipeline. Coxon is no longer an outlier. Turner and Kokotajlo joined him Monday, and the proposed bounty system is explicitly designed to create more of them. Every additional insider who testifies makes the industry’s self-regulation story harder to sustain.

The CEOs didn’t show up. Their critics did. That says plenty about who holds the cards right now.

Frequently asked questions

What did Jacob Coxon say at the NYC AI hearing?

Coxon told the City Council that AI companies are “extremely reckless given the stakes,” said researchers have lost full control of AI systems, and warned the models are approaching the point where they can improve themselves.

What AI laws is New York City considering after the hearing?

Proposals include mandatory “kill switches” for AI tools used in the city, third-party safety vetting before deployment, and a first-in-the-nation cash reward system for whistleblowers, with fines of $25,000 per offense.

Which AI whistleblowers testified at the October 5 NYC hearing?

Jacob Coxon (ex-Anthropic), Alex Turner (ex-Google DeepMind), and Daniel Kokotajlo (ex-OpenAI) testified alongside policy officials from Anthropic, OpenAI, Meta, and Google. None of the companies sent their CEO.

Why did the NYC Council threaten AI companies with subpoenas?

Speaker Julie Menin sent a follow-up letter warning she would use the council’s subpoena power after Google, OpenAI, and Anthropic ignored initial invitations. Meta agreed voluntarily; SpaceXAI was actually subpoenaed.

Sources: Bloomberg News, New York Post, AFP (via NZ Herald), Reuters