
OpenAI, Google, Microsoft and Anthropic Back One AI Training Opt-Out in robots.txt
Four of the biggest AI companies agreed on Friday to stop making publishers beg each of them separately. OpenAI, Google, Microsoft and Anthropic said on Oct. 30 that they will honor a single robots.txt token that lets any website opt out of AI training with one entry, instead of a separate blocking rule for every company’s crawler. Until now, a publisher that didn’t want its articles feeding AI models had to maintain a small pile of bot-specific rules: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, and more besides. Miss one, and that company’s crawler kept feeding. The new token collapses all of that into a single line, __usage_ai-training, written where a crawler name normally goes in robots.txt. ...













