Google, Microsoft, Anthropic and OpenAI plan to support a new website crawling standard that lets publishers block training-data collection for generative AI with a single robots. txt entry. The companies have already submitted a draft proposal to the Internet Engineering Task Force, which maintains the Robots Exclusion Protocol. OpenAI and Microsoft expect to add support in the coming months, while Google will follow next year. For businesses that publish content, this matters because opt-out management could become far less fragmented.
Single entry to block AI training crawlers
Today a publisher that wants to refuse AI training scraping must add a separate robots. txt entry for every AI provider. When one provider operates several scrapers, each scraper needs its own entry, which makes maintenance laborious and error-prone. The proposed standard replaces that patchwork with one instruction covering all scrapers that collect data for generative AI models. The scope is defined narrowly and would not apply to models that perform exclusively non-generative tasks. That distinction separates training for content generation from other forms of machine learning.
The mechanism builds directly on the existing Robots Exclusion Protocol, the standardized format behind robots. txt files. Website operators already use these files to tell crawlers which pages they may access and to limit visit frequency. Search engine management was the original use case, and AI training-data collection became a second major application. Under the proposal, the new AI opt-out becomes part of that same protocol rather than a parallel system. The draft status means technical details can still change before the Internet Engineering Task Force finalizes the format.
The timing reflects pressure from publishers and the rapid growth of training-data scraping alongside search crawling. Robots. txt was designed for relatively simple allow-and-block rules, not for dozens of AI crawlers with different owners and purposes. Microsoft separately described a setting called AI Grounding for cases when a model opens a page to answer a user question rather than to collect training data. Website operators could use it to stop Microsoft services from sending such access requests. The control would also cover third-party AI tools that process prompts with Microsoft technology, according to the company.
What simpler opt-out means for business
For companies that run media sites, documentation portals, marketplaces or corporate blogs, a single blocking entry lowers the cost of controlling AI reuse. Legal and marketing teams will no longer need to track each new crawler name and update files provider by provider. Small publishers without dedicated web operations gain the most, since one line is easier to deploy and audit than a long list. Large organizations with many domains still benefit through faster rollout and fewer inconsistencies across properties. The practical effect should appear in content-management workflows and standard website launch checklists.
The proposal also creates new decisions around visibility, grounding and licensing. Microsoft's AI Grounding distinction shows that blocking training is different from blocking real-time answers that cite a page. OpenAI plans similar controls, which suggests publishers will face separate toggles for training and for answering user queries. Meta has expressed support for the idea but has not committed to implementation, so coverage will remain incomplete at first. Companies should therefore verify which crawlers respect the new entry and keep existing provider-specific rules until support is confirmed.
The marker to watch is formal adoption inside the Robots Exclusion Protocol and activation by OpenAI and Microsoft in the coming months, followed by Google next year. If major crawlers honor the single entry consistently, publisher policies and content-licensing negotiations will start referencing it as the default. Uneven implementation or broad exceptions for non-generative tasks would preserve the current fragmented approach. The next implementation announcements will show which outcome is taking shape.
