The biggest names in AI are backing a new way to opt out of scraping

Judgment

Publisher standards and legal teams must decide whether to rely on the shared robots.txt policy token now, even though Google and Microsoft plan support only next year and Meta has not committed to honoring it. The tradeoff is a simpler single opt-out against uneven, delayed enforcement across crawlers.

Basis

Kevin Kelley, a technical lead for standards at Anthropic, proposed the shared “policy token” signal within the IETF; it would live in a site’s robots.txt file. OpenAI, Google, Microsoft, and Anthropic say they will restrict their AI crawlers from accessing a site if they encounter the shared token. Anthropic has already adopted the signal; Google and Microsoft plan to support it next year; OpenAI says it will implement it in the “coming months.” Microsoft says it will offer granular controls letting publishers opt out of AI training or AI grounding, and notes sites will still appear in Bing search results regardless of preferences. Google says it previously implemented similar controls. Meta communications head Andy Stone signaled support but did not say whether Meta will honor the setting. The mechanism is one publisher preference applied across multiple AI companies instead of separate settings for each.