The New AI Training Opt-Out: What Website Owners Need to Know

Website owners have long faced a frustrating choice: explain their AI training preferences separately to each provider, or accept a patchwork of crawler rules that is difficult to maintain. A proposed shared signal in robots.txt promises a simpler way to communicate that preference.

Simpler does not mean universal, or even identical across providers. The proposal is still an Internet-Draft, support is arriving on different schedules, and a training opt-out can have different consequences for AI-generated answers. The decision is not just whether to add a line to a file. It is which uses of your content you want to permit, and where you still want to be discovered.

Three Different Uses of Your Content

Before changing anything, separate these activities:

• Search indexing: a search engine discovers and indexes pages so they can appear in search results.
• AI training: content is used to modify the learned parameters of a generative AI model.
• Grounding: a system retrieves content when generating an answer so the response can use information from it.

A page can be eligible for search without the owner wanting it used for training. An owner might also want their work cited in AI answers while opting out of training. Those are reasonable preferences, but the controls that express them are provider-specific. Microsoft describes training and grounding separately in its publisher guidance.

What the Shared Signal Looks Like

The IETF Internet-Draft proposes a policy token in the existing robots.txt format. A whole-site training opt-out looks like this:

User-Agent: __usage_ai-training
Disallow: /

The token expresses a preference about a use of content, rather than naming a particular crawler. The draft also permits path-specific rules. Add the relevant group to the existing robots.txt at the root of each applicable host; do not replace your other crawler rules with this example.

This is a proposed preference signal, not a completed standard or a technical barrier that prevents access. It relies on participating providers implementing and respecting it. It does not itself remove previously collected data, settle licensing questions, or ensure every third party follows the preference.

Support Is Planned, Not Instant

The announcements on October 30, 2026 describe different implementation timelines:

• Google aims to support the shared signal in 2027. Its existing Google-Extended control continues to work. Google Search Central
• OpenAI says it aims to implement the signal in the coming months; its existing crawler controls remain in effect. OpenAI
• Microsoft plans to begin honoring its new training and grounding preferences on February 1, 2027, as it crawls pages. Microsoft Bing

Until each provider confirms implementation, keep the provider-specific rules that reflect your current choices. Check for updates before relying on a shared signal alone.

The Google Grounding Trade-Off

Google says that, when the shared training opt-out is present and Google-Extended is not explicitly specified, it plans to treat the preference as a disallow for Google-Extended. That control covers more than the simple phrase “training opt-out” might suggest: Google says explicitly allowing Google-Extended makes sites eligible for grounding in Gemini Apps. Google's planned mapping therefore matters to publishers who want to limit training but still want that Gemini grounding opportunity.

Google also says company-specific controls can operate alongside the shared signal. Review its current documentation and your choices carefully before adding an explicit exception: allowing Google-Extended has its own implications. It would be a mistake to assume that the shared opt-out produces the same training-versus-grounding split everywhere. Google Search Central

Google Search itself has separate controls for snippets and generative features. Its guide to controlling what you share is the right starting point for those decisions. Neither an opt-out nor an explicit allow guarantees a citation or a visit.

Microsoft Separates Training and Grounding

Microsoft plans to treat the shared training preference and its own page-level grounding preference as independent choices. A publisher can disallow training while allowing content to inform supported Microsoft AI answers. Microsoft says these preferences do not change eligibility for traditional Bing search results, subject to the usual crawling and indexing requirements.

Microsoft also offers a grounding option that limits use to its own experiences rather than third-party grounding services. Restricting grounding can reduce visibility in AI-generated experiences; allowing it does not guarantee inclusion, traffic, or payment. The new page-level controls and their exact syntax are documented in Microsoft's announcement.

There is a time-sensitive detail for sites using NoArchive or NoCache to restrict these uses today. Microsoft says those older tags will no longer serve as training and grounding preferences after March 1, 2027. To maintain grounding restrictions, eligible sites must publish the new page-level preference and record a temporary preference in Bing Webmaster Tools by that date. The temporary transition option opens December 15, 2026 for sites using those tags as of that date. Training requires its own new preference; there is no equivalent temporary training setting. Follow Microsoft's full instructions, including its recrawl and expiration conditions.

What About ChatGPT Search?

OpenAI distinguishes its training crawler, GPTBot, from OAI-SearchBot, which it uses for search. Its crawler documentation describes separate controls, so a decision about training should not automatically be treated as a decision to block search discovery. OpenAI also says the new shared signal will coexist with its existing crawler rules.

Check the documented behavior of each crawler and user-initiated access separately. A search-oriented crawler being allowed makes discovery possible; it does not promise that an assistant will retrieve or cite a particular page.

A Practical Checklist for Site Owners

1. Decide what you want. Record separate preferences for training, search indexing, and AI answer grounding. Different sections of a site may warrant different choices.
2. Inventory existing controls. Review robots.txt, page-level meta tags and headers, CDN or firewall rules, and any platform settings. Check each host and important path.
3. Keep current provider rules in place. The shared signal is still being rolled out. Avoid removing a working control merely because the proposal has been announced.
4. Review provider-specific consequences. In particular, assess Google's planned Google-Extended mapping and Microsoft's separate grounding settings. If you use NoArchive or NoCache for Microsoft AI restrictions, plan the 2027 transition now.
5. Verify the published result. Fetch the live /robots.txt file, check the exact token and paths, and make sure your intended search crawlers can still reach important pages. Monitor Search Console, Bing Webmaster Tools, and server or CDN logs for unexpected access problems.
6. Revisit the decision. Rollout dates, provider behavior, and the draft may change. Assign an owner to review the controls after implementation rather than treating the file as a one-time task.

The shared signal is a useful step toward making publisher preferences easier to express. The valuable work is deciding what those preferences mean for your business—and checking that each provider's actual behavior matches the choice you intended.