Google, Microsoft, OpenAI and Anthropic have committed to honouring a shared signal through which website operators can opt out of content collection for AI training. Rather than maintaining separate instructions for each provider, publishers would be able to communicate their preferences through a common setting.
Proposed by Anthropic’s technical lead for standards, Kevin Kelley, the mechanism sits within robots.txt, the file websites use to communicate with automated crawlers. Operators can exclude an entire site or specify exceptions, such as permitting access to a press kit. However, the proposal is an individual submission: the Internet Engineering Task Force has not endorsed it, and it has no formal standing in the standards process.
Implementation schedules differ. Anthropic has begun work on support, while Microsoft has set 1 February 2027 as its start date. Google is targeting early 2027, and OpenAI expects to implement the signal in the coming months.
The draft expresses a preference about content use, rather than imposing a general prohibition on crawling. It cannot undo previous collection, and ordinary Google and Bing search visibility is unaffected. Private agreements between publishers and AI companies can override the preference.
Microsoft is also introducing a separate page-level control for grounding, where an AI answer draws on a live web page. Although restricting grounding can reduce visibility in AI-generated experiences, Microsoft says it will not affect a page’s position in Bing search results. Meta has expressed support for a common signal without committing to honouring it.
