On October 30, Google, OpenAI, Microsoft and Anthropic announced support for a shared way to express AI-training preferences through robots.txt. The proposed __usage_ai-training policy token would let a website owner state a preference once across participating providers instead of maintaining separate instructions for each company. The announcement is significant for publishers and marketers, but it is not a universal switch that is already live: it is based on an Internet-Draft, adoption will be staged, and the signal is about training rather than every way AI systems may use a page. (Search Engine Land; IETF draft)

A shared signal is emerging, not a finished standard

The new draft adds a per-purpose policy token to the familiar robots.txt format. Under a group for __usage_ai-training, a site could express a preference against using all or selected paths to train generative AI models. The proposal changes neither the existing file syntax nor the rules used by ordinary crawler-specific groups. That distinction matters: the draft calls the signal a content-use preference, not a directive that blocks a crawler from retrieving a page. Site operators who need to restrict access still need to manage the relevant crawler rules separately. (IETF Internet-Draft)

The draft was published October 30 and is explicitly work in progress, not a final IETF standard. Its author notes that Internet-Drafts can change or be replaced. The announcements therefore represent a meaningful alignment among major providers, not a guarantee that every AI service will follow the same policy or that the signal will be enforced by law. OpenAI says it aims to implement the shared signal in the coming months; Microsoft says it will begin honoring its new preferences on February 1, 2027; Google says its planned support is for 2027. (OpenAI; Microsoft Bing; Google’s announcement, reported by Search Engine Land)

Training preferences do not automatically control AI search visibility

The biggest operational risk is treating “AI training” as shorthand for every AI use. The IETF draft defines training as using content to modify a model’s learned parameters, and says the preference is independent of traffic-management rules. A training opt-out should not be assumed to stop a search engine from indexing a page, retrieving it for a live answer, or citing it. Those outcomes depend on other controls and each provider’s implementation. (IETF draft)

Microsoft is making that distinction explicit: Bing offers separate page-level preferences for AI training and AI grounding, and says the choices do not change eligibility for traditional Bing results. Google’s announced mapping has a different wrinkle: if a site disallows the shared training token without explicitly specifying Google-Extended, Google plans to treat that as disallowing Google-Extended. Google documents Google-Extended as a control for training and grounding in some of its other systems, while its Search AI features have separate visibility controls and require pages to remain eligible for Search snippets. A blanket rule can therefore have different consequences across platforms. (Microsoft Bing; Google’s interpretation; Google Search Central)

Audit existing rules before changing robots.txt

For businesses, the immediate job is governance, not rushing to paste a new line into a production file. Inventory current robots.txt groups, page-level robot tags, and provider-specific settings. Then decide which content should be available for model training, which should be usable to ground answers, and which should remain discoverable in ordinary search. Public service pages, product documentation and original research may have different commercial and visibility goals; a single site-wide choice may not fit all three.

Pay particular attention to older Bing directives. Microsoft says its new training and grounding preferences will replace NoArchive and NoCache for those uses. Sites relying on those tags must publish the appropriate new preferences; sites that want to preserve a temporary grounding restriction may also need to record it in Bing Webmaster Tools by March 1, 2027. Verify the applicable transition against Microsoft’s instructions before changing anything, and test page-level exceptions carefully. (Microsoft Bing’s transition guidance)

What marketing teams should do now

  • Map the choices: Separate model training, live-answer grounding, and search indexing in your content policy.
  • Protect valuable discovery: Do not block a useful page from AI search simply because you want to limit model training; confirm each platform’s specific behavior first.
  • Document exceptions: Set rules by section or page where possible, and assign an owner to review changes as providers roll out support.
  • Validate after rollout: Check server logs and search visibility, and revisit the policy when the IETF draft and provider documentation mature.

The shared signal could make content-use preferences easier to communicate, but it does not remove the strategic trade-off between protecting assets and being visible where customers now search. Real Internet Sales helps businesses build practical AI-search and content policies that balance both. Call 803-708-5514 or visit realinternetsales.com.