Google, OpenAI and Microsoft let sites opt out of AI training

Robots.txt was built for service folders. Now the same file carries a question that decides the fate of content: may this text train a model? On October 30, Google, OpenAI and Microsoft agreed to answer it the same way.

What they actually agreed to

Nothing shipped. The three said they will support a shared mechanism: a site owner states in robots.txt whether AI training is allowed. The spec is being discussed at the IETF — so this is an attempt to make refusal part of the web, not a thread of legal emails.

Google and Microsoft promise support from 2027. OpenAI is on the list without a date. GIGAZINE nails the practical part: because everyone adopts the same mechanism, sites won't have to argue with each lab separately.

Why this matters more than it sounds

Until now, refusal was politeness. A site wrote «don't crawl», and then: some crawlers listened, some didn't, some listened until the policy changed. For publishers it felt like asking a question and getting silence.

Now refusal becomes a protocol. A protocol is not a request: you either follow it or break it in public. So read this as data access moving from negotiation into infrastructure — not as AI companies getting kinder.

There is a second edge. The mechanism covers permission too. One format means licensing can be automated: a crawler reads the signal and either moves on or pays.

Where it breaks

The date. 2027 is not tomorrow. Another model cycle fits between the announcement and a working protocol — and another round of «we trained on what was available». Sites waiting for 2027 may find their archive already scraped.

Second: the IETF does not stamp specs overnight. While the discussion runs, big labs can ship their own implementations, and the «shared mechanism» becomes a set of dialects. Familiar ending: a standard exists, compatibility doesn't.

Third: who checks compliance? The announcement says nothing about penalties for ignoring the signal. Without consequences this is politeness again, just in new syntax — and the arbiter will be courts, not the spec.

Prediction: By the time support lands in 2027, at least one major lab will offer its own variant of the signal — and the argument returns to where it started: who reads the rules and who writes them.

Disclaimer: the primary source (the October 30 announcement by Google, OpenAI and Microsoft) is not in the registry — based on GIGAZINE and Mynavi News reports, not officially confirmed.

Mynavi News and GIGAZINE