Google, Microsoft, OpenAI and Anthropic will honour a single opt-out from AI training in robots. txt, replacing the need to write separate rules for each crawler. The signal is defined in an IETF draft by Kevin Kelley of Anthropic, published on Friday, with support announced by the other three companies the same day. For business, this matters because content policy for generative AI becomes one setting instead of fragmented controls.
How the shared training signal works
A website adds a group for a new policy token, __usage_ai-training, to its robots. txt file and lists paths excluded from training. The file already tells crawlers what they may fetch, and the draft example blocks an entire site except its press kit. A site with no such group has expressed no preference, while existing rules for all crawlers do not count as a training preference. Private agreements between a site and an AI company can override the token.
The draft describes the token as a stated preference about use, not an order about crawling. Anthropic said the mechanism will complement its existing crawler controls rather than replace them, and OpenAI said controls such as GPTBot will still apply. Google said a site that uses the token without mentioning Google-Extended will be treated as blocking Google-Extended, its own AI training control. Microsoft will honour the new preferences from 1 February 2027, replacing the NoArchive and NoCache tags used since 2023.
The proposal remains an individual IETF submission with no formal standing, and the IETF datatracker says it has not been endorsed. Kelley said implementation will take time across services because complications and edge cases will emerge. OpenAI aims to implement the signal in the coming months, while Google points to early 2027 in its crawler documentation. Meta communications head Andy Stone voiced support for a common signal on X, without saying whether Meta would honour it, according to The Verge.
What this means for companies using AI
For companies that publish documentation, media, product data or knowledge bases, one token lowers the cost of managing AI-training permissions across vendors. Small firms gain because they no longer need to track each crawler name and syntax to express refusal. Large publishers gain consistency across domains, while still keeping separate licensing deals that override the shared signal. The trade-off is operational: rules must be written carefully by path, since the example shows selective exclusion such as keeping a press kit open.
Visibility and licensing limits need separate decisions. Microsoft said the preferences do not establish licensing or payment arrangements, and blocking AI grounding will significantly reduce visibility in AI-generated experiences without changing Bing search ranking. Microsoft is adding a page-level grounding setting that can allow its own products such as Copilot while blocking grounding services sold to other firms. Sites that allow Google-Extended can appear in Gemini app answers and receive clicks, according to Google Search Advocate John Mueller.
The marker to watch is adoption by 1 February 2027, when Microsoft starts honouring the preferences and Google moves toward early 2027 support. If major publishers deploy the token and crawlers respect path-level exceptions in practice, the mechanism will become a standard part of content distribution. Failure would look like uneven crawler behavior or continued reliance on company-specific controls and paid blocking tools such as the approach described by Cloudflare.
