Google, OpenAI, Microsoft and Anthropic have agreed to back one shared AI training opt-out in robots.txt. Instead of blocking each company’s crawler one by one, you would add a single entry that says “don’t use my content to train generative AI.” It isn’t live everywhere yet, and it’s only a draft, so here’s what’s real today.
The quick version, before the details:
- Announced: October 30, 2026, by all four companies.
- What it is: a policy token for robots.txt, filed as an IETF draft.
- Who it’s for: site owners, bloggers and publishers who want to say no to AI training.
- When it works: Anthropic has started, OpenAI says “coming months,” Microsoft from February 1, 2027, Google in 2027.
- Catch: it’s a request, not a lock. Nothing forces a crawler to obey.
If you run a site, the one-line takeaway is that your robots.txt may soon do more work than it does now.
What the new AI training opt-out actually does
Right now, blocking AI training means keeping a list. GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and so on. Miss one, and that company’s crawler may keep reading your pages.
The new signal replaces that guesswork. According to SiliconANGLE, one entry would cover scrapers that collect data to train generative AI models. Models that only do non-generative jobs aren’t included.
The token is called __usage_ai-training. The Next Web reports that you add it as a group in robots.txt, and the draft’s own example blocks a whole site except its press kit. Check the draft for exact syntax before you copy anything from a blog post, including this one.
The draft is called “Machine-Readable Training Preference Signal.” It was submitted to the IETF by Kevin Kelley of Anthropic, and the IETF hasn’t endorsed it yet.
Who honors it, and when
Each company is moving at its own speed. The table below pulls together what has been reported so far.
| Company | Status | Notes |
|---|---|---|
| Anthropic | Work has started | Full support across services will take “some time” |
| OpenAI | “Coming months” | Existing controls like GPTBot still apply |
| Microsoft | From February 1, 2027 | Replaces its older NoArchive and NoCache tags |
| Planned for 2027 | Treats the token as a Google-Extended block if you haven’t set that yourself |
Meta hasn’t signed on. Its communications head, Andy Stone, said he supports a common signal but didn’t say Meta would honor this one.
What this opt-out doesn’t do
This is the part many headlines skip. The signal states a preference about training. It is not a firewall, and it doesn’t touch several things you might assume it does.
Here’s what the reporting says falls outside it:
- It doesn’t stop crawling. It says how content may be used, not whether a bot can visit.
- Private deals between a site and an AI company can override it.
- It doesn’t set up licensing or payment. Microsoft says so directly.
- Blocking it doesn’t change your Bing ranking, according to Microsoft.
The takeaway: opting out is a statement of intent, and its value depends on companies keeping their word.
AI answers are a separate matter. Microsoft has an “AI grounding” setting for pages used to answer a user’s question, and it warns that blocking it can reduce your visibility in AI-generated results. Google’s Gemini grounding works in a similar spirit, per Unite.AI.
Should you add the signal to your site?
That depends on what you want from AI. If you’d rather not feed models with your writing, the signal is a cheap way to say so once it’s supported. If you want to show up in AI answers, think twice before blocking everything.
Visibility in AI products is already shifting. Google is putting ads into AI answers, which we covered in Google AI Mode Ads Are Going Global. And some AI tools can’t open certain sites at all, as we explained in our look at OpenAI Aura blocking sites.
A sensible plan is to wait for your own platform’s support, then test. Your CMS or SEO plugin will likely add a toggle once the draft settles. Until then, keep your existing bot rules in place, since they still work.
If you manage a robots.txt file by hand, now is a good time to tidy it. Remove dead entries, add comments explaining each rule, and keep a copy of the file so you can roll back if a change blocks something you didn’t mean to.
Frequently Asked Questions
Is the AI training opt-out an official standard yet?
No. It’s an individual IETF draft, not an endorsed standard. The companies plan to fold it into the Robots Exclusion Protocol, but there’s no adoption deadline.
Will it hide my site from Google Search?
Not according to current reporting. Microsoft says it won’t change Bing rankings. Reports don’t say whether Google Search is affected, so watch Google’s documentation.
Do I need to remove my old robots.txt rules?
No. OpenAI says controls like GPTBot still apply, and a site with no token group has expressed no preference under the new system. Keep what you have.
Does it work for Meta’s AI?
Not yet. Meta voiced support for a shared signal but hasn’t committed to honoring this one.
Our take: worth adding, just don’t expect miracles
One line instead of a growing blocklist is a real improvement, and having four rivals agree is rare. But with a draft status, staggered dates and no enforcement, treat it as a polite sign on the door. Add it when your tools support it, keep your current bot rules, and decide on AI answers separately.
Leave a Reply