Major AI and search companies have agreed to support one shared way to tell crawlers whether your website may be used to train AI models. The signal lives in your robots.txt file. Most small business sites do not need to change anything today, but if you use Bing's NoArchive or NoCache tags, a deadline applies.
What changed
On October 30, 2026, Google announced plans to support a shared AI training control in robots.txt, built on work from the AI Preferences working group at the Internet Engineering Task Force (IETF). Instead of setting rules for each AI company, a site owner adds a shared "policy token" to opt out of training across participating providers. The proposal is an individual Internet-Draft written by an Anthropic standards lead. The IETF has not endorsed it, so it is not a final standard.
Microsoft announced the same day that it will honor the shared training preference and add a separate grounding preference. Grounding means using your pages to inform an AI-generated answer. Microsoft's settings are:
- AI Training: set in robots.txt for a site or path, or on an individual page. A page-level setting overrides robots.txt.
- AI Grounding: set per page, with Allow, DisallowExtended or Disallow. DisallowExtended limits use to Microsoft's own experiences and excludes grounding services it provides to third parties.
Both settings default to Allow if you publish nothing. Microsoft says the choices do not change whether you are eligible for regular Bing search results, though disallowing grounding means reduced brand visibility in AI experiences. Google says its existing Google-Extended control keeps working alongside the new signal. If a site opts out with the shared token and does not mention Google-Extended, Google plans to treat that as disallowing Google-Extended. The Verge reports that Anthropic already supports the signal and that OpenAI plans to follow in the coming months.
What it means for your business
If you do nothing, your content stays eligible for AI training and grounding, as it is now. The new signal gives you one place to opt out if you want to, and it does not affect how you rank in classic search results.
The deadline hits sites that use NoArchive or NoCache to control Bing. Microsoft will start honoring its new preferences on February 1, 2027. After March 1, 2027, those older tags will no longer be read as a training or grounding preference. Google says it aims to support the shared control in 2027.
What to do now
- Decide where you stand. Do you want AI tools to learn from your content, quote it in answers, both or neither?
- Open your robots.txt file at yourdomain.com/robots.txt and note any rules for AI crawlers or Google-Extended.
- Check your page templates for NoArchive or NoCache tags. If you find them, plan to replace them with Microsoft's new preferences before March 1, 2027.
- If you want to stay visible in AI answers, leave grounding on Allow. Opting out can reduce how often your brand appears there.
- Wait for Google to publish its exact implementation steps before you add the shared token to a live site.
The details
The draft describes the token as a robots.txt group keyed in the User-agent line, with no new directives. Because it is an early draft that expires on May 3, 2027, the syntax could change.
Our free SEO audit shows whether a page can be indexed, and our SEO team can review your robots.txt rules. For more on how AI is changing search, read our note on Google bringing ads to AI Mode in 10 new countries.