Google, Microsoft, OpenAI, and Anthropic have committed to supporting a shared signal for AI training preferences. Website owners currently need separate rules for each company's automated collection programs, known as crawlers. The proposal would let owners communicate their choices to several companies with one rule.
The signal goes in robots.txt, a file websites use to give instructions to crawlers. Owners can express a preference against training use across an entire site while allowing selected sections. The change does not remove pages from regular search results, and it cannot undo content collection that has already happened.
The companies have different schedules for introducing support. Anthropic has begun work, while OpenAI plans to implement the signal in the coming months. Google expects to support it in early 2027. Microsoft has set February 1, 2027, as its start date.
Anthropic's Kevin Kelley wrote the proposal, which is under consideration at the Internet Engineering Task Force. It has not received the organization's endorsement or formal standing in its standards process. The signal expresses a preference about content use rather than directly preventing crawling. Its effect depends on companies honoring it.
