A New Robots.txt Signal Lets Websites Opt Out of AI Training

If your company has a website, you already have an opinion on whether AI models should learn from it, even if nobody has written that opinion down. On October 30, OpenAI said it will start honoring a new, shared way for site owners to state that preference in one place: robots.txt, the plain text file almost every website already has.

This isn't a new idea from one company. It's built on a proposal called the "Machine-Readable Training Preference Signal," filed as an individual Internet-Draft with the IETF on the same day by Kevin Kelley at Anthropic, who said Anthropic has already started implementing support for it. According to coverage from Search Engine Roundtable and Unite.AI, Google and Microsoft have separately said they plan to support it too, though on their own timelines. For a marketing, content or IT team in India, this matters more than it sounds: it's the first time multiple large AI companies have agreed on one technical signal for "don't train on this," instead of each running its own separate rule.

What's actually new here

Until now, if you wanted to stop an AI company from training on your site, you had to disallow its specific crawler in robots.txt: GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended for Google, and so on, one line per company, with no shared meaning across them. Every vendor read its own rule its own way.

The new draft defines a single, cross-vendor policy token, __usage_ai-training, that you attach to a User-agent group in robots.txt. A site can pair a blanket disallow for that token with an allow for specific paths it's happy to have used, for example its own press kit. Written once, in theory, it tells every AI company that adopts the standard the same thing, instead of you maintaining a separate list of crawler names that changes every time a new AI company shows up.

This is still an individual draft, not a finished IETF standard, and support is voluntary. Google's own documentation points to an early-2027 timeline for using the signal, alongside its existing Google-Extended control. Coverage of Microsoft's plans describes it extending the preference down to the page level through meta tags and HTTP headers, with a separate "page-level grounding preference" for how a model can reference your content in an answer, distinct from training on it. None of this replaces what already exists: OpenAI has said its current robots.txt handling, including GPTBot, continues to apply exactly as before.

Training, search and "grounding" are three different questions

It's worth being precise here, because the three get mixed up constantly in business conversations. Disallowing GPTBot already tells OpenAI not to use your content to train future models. Separately, disallowing OAI-SearchBot means your pages won't show up as answers in ChatGPT's search feature, though you can still appear as a plain navigational link. The new preference signal is aimed specifically at the training question, across companies, in one place. Whether a model can reference your live page when answering a question is a related but separate setting, and Microsoft's "grounding" control suggests vendors will keep that one distinct. Don't assume one line in robots.txt now controls all three.

Why "block everything" isn't automatically the right call

The instinct for a lot of Indian businesses will be to disallow AI training everywhere and move on. Think about that before you do it. If your buyers are increasingly asking ChatGPT, Gemini or Perplexity to recommend products and services in your category, as our checklist for retail and marketing teams on this site goes into, being absent from AI training can mean being absent from AI answers. For a services firm, a course provider or a B2B vendor, showing up accurately when a prospect asks an AI assistant "who does X in India" can be worth more than the content protection you get from opting out.

There's no single right answer. A fair way to think about it: separate your content into categories before you decide.

1. Marketing and product pages you want AI assistants to represent accurately when someone asks for a recommendation. Consider allowing these.
2. Proprietary research, paid content or client work you don't want repurposed into someone else's answer for free. These are reasonable candidates to disallow.
3. Pages with personal data or anything under a client NDA. These should already be access-controlled, not just signalled in robots.txt.

What to actually do this week

1. Pull up your current robots.txt. It's at yourdomain.com/robots.txt. Most teams have never looked at it since whoever set up the website wrote it once and left.
2. List what each AI crawler is currently allowed or blocked from doing, per company, using the existing crawler names (GPTBot, ClaudeBot, Google-Extended, OAI-SearchBot and similar). This is the part that works today, regardless of the new draft's timeline.
3. Decide your policy by content type, using the three categories above, and write it down as an actual decision, not a default.
4. Bring in whoever owns content and legal, not just IT. Whether your content can be used to train a competitor's model is a business call, and it shouldn't be made by whoever happens to edit the website.
5. Put a date on your calendar for early 2027, when Google's documented timeline for the shared signal is expected to apply, and revisit the decision once the companies that matter to your traffic have actually shipped support.

The honest caveats

This is a voluntary standard, not law. Compliance depends on each AI company choosing to honor it, and the draft itself notes that private agreements between a site and an AI company can override the public preference anyway, which matters if you've ever signed a content licensing deal with one. Enforcement, auditing and what happens when a company doesn't honor its own stated policy are all open questions nobody has answered yet. Treat this as a useful new lever, not a guarantee.

If you're deciding how your organization should handle AI vendor relationships more broadly, not just this one setting, that's a good conversation to have with a partner who's implemented AI policy end to end. Our AI consulting work with Indian businesses starts exactly there: a short audit of where your data already is, who can see it, and what you want AI systems to do with it before you write the policy down.