Signals · Hot · first seen October 30, 2026
OpenAI, Google, Microsoft and Anthropic back one robots.txt line to opt a site out of AI training
On Oct. 30, the four companies said they will honor one shared robots.txt entry that tells AI companies not to train models on a site's pages. It comes from a proposal at the IETF, the internet standards body: site owners add a user agent named __usage_ai-training with Disallow: /. Anthropic already honors it; Google and Microsoft plan to start in 2027 and OpenAI in the coming months, Engadget reported.
Why it matters to a business like yours
Until now, keeping your website out of AI training meant a separate rule for each company's crawler, and one line now covers all four without taking you out of search, though only for crawling from here on. Decide whether you want your pages used for training, and if not, ask whoever runs your site to add the line.
Sources
- Supporting a shared AI training control for websites (Google Search Central, October 30, 2026)
- Google joins OpenAI and Anthropic in promising new scraping controls (Engadget, October 31, 2026)
- Anthropic, Google, Microsoft, and OpenAI Agree to Let Sites Opt Out of AI Training (Thurrott, October 30, 2026)
On the Insider
- OpenAI, Google, Microsoft and Anthropic will honor one "don't train on my site" line (October 31, 2026)
History
- October 31, 2026: Hot