Four of the biggest AI companies agreed on Friday to stop making publishers beg each of them separately. OpenAI, Google, Microsoft and Anthropic said on Oct. 30 that they will honor a single robots.txt token that lets any website opt out of AI training with one entry, instead of a separate blocking rule for every company’s crawler.
Until now, a publisher that didn’t want its articles feeding AI models had to maintain a small pile of bot-specific rules: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, and more besides. Miss one, and that company’s crawler kept feeding. The new token collapses all of that into a single line, __usage_ai-training, written where a crawler name normally goes in robots.txt.
That’s the pitch. The reality is messier, slower, and still voluntary.
How the AI training opt-out token works
A site owner adds a block to robots.txt that looks like this:
User-Agent: __usage_ai-training
Disallow: /
That tells all four companies not to use the site’s pages for training generative AI models. Adding Allow: lines underneath carves out exceptions for specific sections, so a publisher can block training on its news archive while leaving, say, its public press releases open.
One distinction matters a lot here. The token controls training permission, not crawling. As the draft behind it puts it, “this policy token is not a directive for crawling behavior.” Bots can still fetch a page that carries the token; they just aren’t supposed to train on it. Google says a site using the shared token will be treated as if it had blocked Google-Extended, its existing training control, unless Google-Extended is set directly.
For publishers, the practical advice right now is boring but important: keep the old per-company rules in place. The new token does nothing at most of these companies yet, and the company-specific directives remain what actually works today.
Who is doing what, and when
Timelines are uneven, and the gap between the earliest and latest adopter is more than a year.
Anthropic has already started implementing support, which makes sense: the proposal comes from its own standards lead, Kevin Kelley, who authored the Internet-Draft behind it. OpenAI says it “aims to implement the signal in the coming months” and published a post titled “Making it easier for websites to express AI training preferences” on the same day. Microsoft gave the firmest date of the four, saying its Bing crawler will start honoring the preferences as it crawls from February 1, 2027. Google said only that it aims to support the token, along with other AI providers, in 2027, though its crawler documentation narrows that to early 2027.
Microsoft went further than the others on granularity. It says it will offer separate controls for opting out of AI training and AI grounding, the process of pointing a model at a verified source to cut down on hallucinations. It also says opting out of training will not affect a site’s eligibility for regular Bing search results, and Google says the same about Google Search: the token is not a ranking signal.
What the token doesn’t do
Start with the paperwork. The document behind the announcement is an individual Internet-Draft titled “Machine-Readable Training Preference Signal,” filed with the Internet Engineering Task Force on Oct. 30 and set to expire on May 3, 2027 unless it is revised or adopted. An individual draft has no formal standing in the IETF process and no endorsement from the body itself. It is a proposal with four big backers, not a standard.
Then there’s enforcement, or rather the lack of it. Robots.txt has never had a mechanism to force compliance. It works because crawler operators choose to honor it, and the commitments in this announcement carry weight only to the extent each company actually implements them. Publishers learned that lesson the hard way in earlier disputes over crawlers ignoring declared preferences, and they will need to verify the token is honored in practice rather than taking Friday’s announcement on faith.
And the token can’t rewind anything. Updating robots.txt does not remove material already collected into existing datasets or trained into existing models. For publishers whose archives were scraped years ago, this is a door closing going forward, not a refund.
The one big name missing
Meta is the conspicuous absence. The Verge reported that Meta supports the idea of a common signal but has not committed to implementing it, and Meta communications head Andy Stone called a shared signal an important step without saying the company will honor the token. That matters because Meta runs one of the largest crawling operations on the web.
Meta is also on a different track entirely. Starting Oct. 30, business accounts consent to Meta scraping their websites if they keep using Facebook, Instagram or WhatsApp, according to Nieman Lab’s reporting from mid-October. So while four rivals announced a shared off switch, Meta moved in the opposite direction for its own business users.
Why this matters
This is the first time the major Western AI labs have coordinated on a single publisher control instead of each shipping its own incompatible version, and that coordination is the real news. Publishers have spent three years fighting on two fronts at once: a legal one, including the New York Times lawsuit against OpenAI and Microsoft filed in December 2023 over millions of articles allegedly used without permission, and a technical one, maintaining a growing list of bot rules that only specialists knew how to write.
A single token, if it sticks, lowers the cost of saying no to something closer to zero. Small publishers and independent sites that never had a lawyer or an SEO team get the same one-line protection as a national newspaper. That’s a genuine shift in who can realistically control their content.
But don’t mistake a shared promise for a solved problem. Nothing here is enforceable yet, nothing is live at three of the four companies, nothing reaches back into models already trained, and the biggest crawler of all hasn’t signed on. Watch three things: whether Meta commits, whether the draft becomes a real IETF standard before it expires in May 2027, and whether publishers can actually verify their opt-outs are being honored. Until those answers land, the token is best read as a truce both sides can live with, not a settlement.
Frequently asked questions
What is the new AI training opt-out?
It is a single robots.txt policy token, __usage_ai-training, that tells OpenAI, Google, Microsoft and Anthropic not to use a site’s content to train generative AI models. One entry replaces the separate per-company blocking rules publishers use today.
How do I add the AI training opt-out to my site?
Add a robots.txt block with User-Agent: __usage_ai-training followed by Disallow: / to opt out of training entirely, and use Allow: lines to carve out exceptions for specific pages. Keep your existing per-company rules in place until each company confirms support.
When will each company honor the new opt-out token?
Anthropic has already started implementing it, OpenAI says it will follow in the coming months, Microsoft will begin honoring it on February 1, 2027, and Google aims to support it in 2027.
Does opting out remove my content from AI models?
No. The token applies only to future training and does not remove content already collected into datasets or baked into existing models. It also does not stop crawlers from fetching pages or affect search rankings.
Sources: The Verge, AI Weekly, AFP.