Google, Microsoft, OpenAI, and Anthropic have pledged to honor a shared signal allowing website publishers to express their AI training preferences. Rather than maintaining separate instructions for each provider's crawler, publishers could communicate their choices through a common policy token in robots.txt, the file used to guide automated website access.
The proposal allows publishers to exclude an entire site from training use while making exceptions for selected sections, such as a press kit. It does not remove pages from regular search results or reverse earlier content collection. Crucially, the token expresses a preference about how content should be used; it is not itself a command prohibiting crawling.
Implementation schedules vary. Anthropic has begun work on support, while OpenAI aims to introduce it in the coming months. Google expects implementation in early 2027, and Microsoft will honor the preferences beginning February 1, 2027. Anthropic says the signal will complement its existing crawler controls, and OpenAI says its current controls will remain applicable.
Written by Anthropic standards technical lead Kevin Kelley, the draft is an individual submission to the Internet Engineering Task Force. The organization has neither endorsed it nor granted it formal standing in the standards process. Under the draft, a website without the relevant group expresses no preference, and private agreements between publishers and AI companies can override the signal. Microsoft also stresses that the preferences do not create licensing or payment arrangements.
