Google, Microsoft, OpenAI and Anthropic Back One robots.txt Line to Opt Out of AI Training

Four of the biggest names in AI agreed on October 30 to read the same two lines in a website’s robots.txt file as a “no” to AI training. Google, Microsoft, OpenAI and Anthropic each said they will honor a single shared policy token, __usage_ai-training, so a site owner can state the preference once instead of writing a separate rule for every lab. The Verge first reported the deal, and TechRadar picked up the story this week as publishers started asking what it actually does.

Until now, opting out meant keeping a list. Google offered Google-Extended. OpenAI named GPTBot. Anthropic named ClaudeBot. Each company read only its own control, and a site that wanted out of all of them had to maintain a rule for each crawler. One shared token replaces that pile with a single group in a file every site already has.

The idea comes from Kevin Kelley, a technical lead for standards at Anthropic. He submitted it to the Internet Engineering Task Force on October 30 as an Internet-Draft titled “Machine-Readable Training Preference Signal.” Google, OpenAI and Microsoft announced their support the same day. Anthropic says it has already started implementing the signal across its services.

How the robots.txt AI training opt-out works

A site that wants out of training for everything writes two lines at the root of its robots.txt file:

User-Agent: __usage_ai-training
Disallow: /

That’s it. A site that wants to keep one corner open adds an Allow: line, and the draft’s own example leaves a /press-kit/ folder in. Path matching follows the same longest-match rules as the rest of robots.txt, which the IETF formalized as RFC 9309 in September 2022.

Nothing about the file format changes. The draft adds no new directives and no new grammar, which is a deliberate choice: existing parsers and existing site tooling can handle it with minimal work.

Two details decide whether a robots.txt file already says anything. A wildcard group, User-Agent: *, never matches the token, so a site that blocked everything in the past has not opted out in this sense. And a file with no __usage_ai-training group at all expresses no training preference, whatever its other rules say.

What each company actually promised

The four commitments are not identical, and the gaps matter.

Anthropic moved first. Kelley wrote on the IETF working group’s mailing list that the company “has already started implementing support” for the token, while adding that “it’ll take us some time to add support across our services.” Rules that name its training crawler, ClaudeBot, take precedence over the shared signal.

OpenAI posted a short statement saying it will honor “a common signal” for foundational generative AI training and aims to implement it “in the coming months.” It gave no firm date. Its existing crawler controls stay in force in the meantime, and its wording is narrower than the draft’s definition of training, which covers changing a generative model’s learned parameters.

Microsoft set the firmest date. It will honor the shared training preference as Bingbot recrawls pages starting February 1, 2027. It is also adding a separate AI Grounding preference, set page by page through a meta tag or an HTTP header, that decides whether a page can feed the answers Bing, Copilot and Microsoft 365 generate on the spot. From March 1, 2027, the older NoArchive and NoCache tags stop counting as training or grounding controls.

Google aims for early 2027, according to its crawler documentation, and its Search Central post by search advocate John Mueller says a site that opts out through the shared line without naming Google-Extended separately will be treated as if it had disallowed Google-Extended. That switch carries a side effect worth knowing: Google-Extended covers both training future Gemini models and grounding in the Gemini apps, so a shared opt-out also takes a site out of Gemini’s grounded answers.

What the signal doesn’t do

The limits are written into the draft itself, and they are blunt.

First, it covers training only. The draft states plainly that the token “is not a directive for crawling behavior.” A crawler that respects it still requests the page. Search keeps working. An AI assistant can still read a page live to answer a question. Publishers hoping for a single switch that hides their site from AI entirely will not get one here.

Second, it looks forward, not back. Content that went into a model before a site adds the token stays there. The draft says nothing about material already collected, and no company has promised to remove it.

Third, it is a request, not a lock. Like everything in robots.txt, it depends on companies choosing to follow it. There is no enforcement mechanism, and the document behind it is an individual Internet-Draft with no formal IETF standing. The IETF datatracker carries a notice to that effect, and the draft expires on May 3, 2027 unless it is revised or adopted.

Meta is the notable absence. Meta communications head Andy Stone told The Verge, “We believe creating a common signal that conveys such preferences to multiple providers at once is an important step,” but the company did not say whether it would honor the token. Amazon and Apple are absent too. A publisher that opts out through the new token still needs the old per-company rules for anyone who hasn’t signed on, and for the months until each of the four ships support.

Why this matters

Publishers have spent three years asking for a way to say no to training without maintaining a spreadsheet of crawler names. This is the first time the four biggest labs have agreed on the same answer, in the same file, on the same day. The mechanism is boring on purpose: two lines, no new grammar, no new tooling. Boring is what makes it adoptable.

Whether it sticks depends on what happens before May 3, 2027, when the draft expires. If the IETF’s AI Preferences working group turns it into a real standard and Meta joins, it becomes the default way the web talks to AI companies about training. If the rollout drags and the draft lapses, publishers are back to one rule per company.

For now, the practical advice is simple. Add the two lines, keep the old per-company rules in place until each lab ships support, and don’t mistake the token for a wall. It tells four companies what you prefer. It doesn’t make them obey, and it doesn’t reach the ones who never agreed.

Frequently asked questions

What is the robots.txt AI training opt-out?

It is a shared policy token, __usage_ai-training, that a site owner adds to robots.txt to state a preference against generative AI training. Google, Microsoft, OpenAI and Anthropic have all said they will honor it.

How do I opt out of AI training with robots.txt?

Add a group to your robots.txt file with User-Agent: __usage_ai-training on one line and Disallow: / on the next. Add Allow lines to carve out folders you want to keep open, such as a press kit.

Does the robots.txt AI training signal block crawling or search?

No. The signal only states a preference about training. Crawlers can still visit your pages, search rankings are unaffected, and AI assistants can still read a page live to answer a question.

When will Google, Microsoft, OpenAI and Anthropic honor the signal?

Anthropic has already started implementing it. OpenAI says it will follow in the coming months. Microsoft starts February 1, 2027, and Google aims for early 2027.

Sources: The Verge, TechRadar, AI Weekly, The Next Web.