OpenAI, Google, Microsoft and Anthropic adopt a shared robots.txt signal for AI training preferences
Four of the biggest AI companies, OpenAI, Google, Microsoft and Anthropic, announced they will adopt a shared standard letting website publishers express their AI training preferences once, instead of configuring each company separately. The signal, proposed by Kevin Kelley, a technical lead for standards at Anthropic, through the Internet Engineering Task Force (IETF), will live inside the familiar robots.txt file of a website, in the form of a "policy token". The four companies commit to blocking their crawlers from a page if they encounter this signal, while Microsoft adds a separate control between model training and "AI grounding", the use of content as a verified source to reduce model hallucinations. Meta voiced support for the idea without committing to honor it. Anthropic has already adopted the signal, Google and Microsoft plan to support it next year, and OpenAI says "in the coming months". For any business or content creator in Greece, the move means that for the first time there will be a way to say "no" to all major AI providers at once, instead of chasing a separate setting with each one.
Sources
This summary was written automatically with AI from the public sources listed above and checked by an independent AI reviewer before publication. For full details, read the original source.