Home / Technology / AI Giants Agree: Publishers Control Web Scraping
AI Giants Agree: Publishers Control Web Scraping
31 Oct
Summary
- Major AI firms will adopt a shared signal for web content scraping opt-outs.
- The 'policy token' will integrate into websites' robots.txt files.
- Companies commit to respecting publisher preferences on AI training data.
Leading AI developers have committed to a unified approach for publishers to manage web content scraping for AI training. OpenAI, Google, Microsoft, and Anthropic will implement a shared "policy token" within a site's robots.txt file.
This token will allow publishers to consistently opt out of data scraping by AI crawlers. Previously, publishers had to manage these preferences individually with each company. This initiative addresses growing publisher concerns about unauthorized scraping and the use of their content in AI models.
While Anthropic has already adopted the signal, Google and Microsoft plan support by next year, with OpenAI implementing it in the coming months. Microsoft will also offer granular controls for opting out of AI grounding specifically.
