To opt out of AI training, add two lines to robots.txt: "User-Agent: __usage_ai-training" and "Disallow: /". It's a cross-provider signal Google plans to honor in early 2027. Until then — and for any provider you want honoring it now — also use per-company training tokens such as Google-Extended, GPTBot, and ClaudeBot. Opting out doesn't affect Search or AI Overviews, but grounding citations in Gemini may stop; to keep them, explicitly set Google-Extended to "Allow: /".
What Changed: One Line for an AI Training Preference
Until now, opting out of AI training meant listing each company's token in robots.txt — Google-Extended (Google), GPTBot (OpenAI), ClaudeBot (Anthropic), and so on. Every new token had to be added, and a missing one meant that company never saw your preference.
On October 30, 2026, a draft using the shared name "__usage_ai-training" was submitted to the IETF working group on AI usage preferences. The same day, Google announced on the Search Central Blog that it plans to support this shared line in early 2027. The draft is still an individual submission, not a standard.
Three points. (1) It's written like any robots.txt group: the name in User-Agent, paths with Disallow and Allow. (2) "User-agent: *" does not count as the shared line. (3) It isn't a crawl directive; it states whether retrieved content may be used for training. To stop crawler visits, keep using each crawler's own name.
How to Write It: The Shared Line and Per-Company Tokens for Now
Don't use the whole site for training
User-Agent: __usage_ai-training
Disallow: /Allow only some pages (same shape as the draft's example)
User-Agent: __usage_ai-training
Disallow: /
Allow: /press/Path rules are the usual robots.txt ones: the longer match wins. Here only /press/ is marked as allowed and everything else as not. You can also list the shared name in the same group as other tokens like GPTBot.
Until providers support it: use training tokens
# Per-company tokens until providers support the shared line (training tokens only)
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /These are training tokens only. Blocking search tokens such as OAI-SearchBot (ChatGPT search) or Claude-SearchBot also stops citations from that AI search. To tell training and search crawlers apart, see how to check whether AI crawlers are blocked.
How Google Treats It: Search Unchanged, Watch Gemini Citations
- If you opt out with the shared line and don't specify Google-Extended, Google treats it as a Google-Extended disallow
- If Google-Extended is explicitly specified, Google follows that. Sites that allow Google-Extended stay eligible to appear in Gemini Apps answers and receive clicks via grounding, even with the shared opt-out
- Google-Extended controls both Gemini training and grounding (using Search content in answers in Gemini Apps and elsewhere). It doesn't affect inclusion or ranking in Search
This is where the decision lies. Once Google supports it, opting out with the shared line alone also refuses Gemini citations as far as Google is concerned. For a site growing AI referrals as an entry point, that can cost you. Today's tokens offer no way to refuse training while keeping Gemini citations, so for Google you choose between allowing both and refusing both.
Opt out everywhere, but allow both for Google
User-Agent: __usage_ai-training
Disallow: /
User-agent: Google-Extended
Allow: /AI Overviews and AI Mode are separate from Google-Extended. Both are part of Search, so they follow your regular Googlebot rules, Search Console's generative AI control, and nosnippet. For which control fits which goal, see how to check noindex.
By Goal: What to Write
| Goal | Write | What happens |
|---|---|---|
| Don't use any of it for AI training (keep search and AI search citations) | Shared line with Disallow: /. For providers you want honoring it now, also Disallow their training tokens | Search and AI Overviews unchanged. Grounding citations in Gemini Apps may stop |
| Opt out of training but keep Gemini citations | Shared line with Disallow: /, plus Google-Extended with Allow: / | You allow Google both training and grounding. Today's tokens offer no way to refuse only training |
| Allow training on some pages only (press releases, etc.) | Shared line with Disallow: / and Allow: /press/ | The longer path wins, as usual in robots.txt. Only /press/ is marked as allowed |
| Stay out of AI search answers | Opting out of training isn't enough. Google: the Search generative AI control or nosnippet; others: their search crawlers (OAI-SearchBot, etc.) | Blocking a search crawler also stops citations from that AI search |
| Keep AI crawlers away | Disallow each company's crawler name (the shared line isn't a crawl directive) | With the shared line alone, crawlers keep coming. Check WAF/CDN blocks separately |
The draft says the preference that applies to a page is the one in effect when the page was retrieved. It works from retrievals after you add it; nothing reaches back to content already retrieved. It also allows private agreements to override these preferences.
Measured: How This Site's robots.txt Reads
To see where your site stands, search robots.txt for the shared line and Google-Extended with the command below. No output means no preference expressed, and Google-Extended follows your "User-agent: *" group.
curl -s https://example.com/robots.txt | grep -inE -A3 'user-agent: *(__usage_ai-training|google-extended)'On November 2, 2026, we read the robots.txt of this site and of a web-developer media site we also run, using the draft's rules ("*" doesn't count; the longer path wins).
| Site | Shared line | Google-Extended | Call |
|---|---|---|---|
| This site (Direbase) | Not set (no preference) | No dedicated group; the "User-agent: *" group applies and allows public pages | Training and citations both allowed. Nothing changes when Google adds support in 2027 |
| A web-developer media site we also run | Not set (no preference) | Google-Extended explicitly set to Allow: / | Even with the shared line added, the explicit Google-Extended rule takes precedence, keeping Google citations |
Neither site opts out. Both are growing AI referrals as an entry point, so our call is to keep allowing training and citations. The media site in particular names Google-Extended explicitly, so even if someone later adds a blanket opt-out with the shared line, Google citations stay. The explicit line acts as a guard against an unintended blanket opt-out.
Conversely, a site that wants to opt out but explicitly allows Google-Extended won't change anything for Google by adding the shared line. Before adding it, check it doesn't conflict with your existing per-company rules. Since November 2, 2026, Direbase's AIO Readiness Check shows whether the shared line is set (against / in favor / no preference) in its robots.txt AI crawler item. It isn't scored.
Common Mistakes
- Disallowing everything under "User-agent: *". That isn't a training preference, and it blocks Googlebot too, removing you from Search
- Assuming the shared line already works. Google plans support for early 2027; for providers you want honoring it now, add their own tokens too
- Thinking blocking Google-Extended takes you out of AI Overviews. Google-Extended doesn't affect Search or AI Overviews
- Expecting the shared line to stop crawlers. The draft states it isn't a crawl directive
- Mixing up GPTBot and OAI-SearchBot. GPTBot is for training; OAI-SearchBot is for ChatGPT search. Blocking the search one removes you from ChatGPT search results
For robots.txt in general and how Disallow works, see how to write robots.txt.
Related Tools
Sources (checked November 2, 2026)
- Google Search Central Blog — Supporting a shared AI training control for websites (2026-10-30)
- Google — Google-Extended (training and grounding, no effect on Search, shared signal support planned for early 2027)
- IETF Internet-Draft — Machine-Readable Training Preference Signal (2026-10-30, individual submission)
- RFC 9309 — Robots Exclusion Protocol
- OpenAI — Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User)
FAQ
What do I put in robots.txt to opt out of AI training?▾
Will opting out of training remove me from Google Search or AI Overviews?▾
What happens to citations in Gemini?▾
If I Disallow everything under "User-agent: *", have I opted out of training?▾
Does it remove data that was already used for training?▾
If both the shared line and Google-Extended or GPTBot rules exist, which wins?▾
Check AI Crawler Access and Your Training Preference Together
Enter a URL to see whether AI search crawlers are blocked in robots.txt or by a WAF, and whether the shared line opts you out of training — all at once.
Run the AIO Readiness Check
Written by
今井政和SEO Director / Frontend Developer
SEO Director with 20+ years of web industry experience. Creator of Direbase and the official WordPress plugin "ORECTIC SEO CHECK." Author of a book on web strategy inspired by Edo-era merchant principles.
@imai_directorRelated Articles
AI Search Is a Selection Layer | The Four-Layer GEO Model and the Layer That Breaks First
GEO works in one order only: Access, Orientation, Understanding, Quotability. Most sites fail where crawlers can't reach them, yet nearly all investment goes into content. A practitioner's reading of The GEO Readiness Manual.
Does llms.txt Work as a GEO Strategy? | CDN Log Data Analysis
Does deploying llms.txt get you cited in AI search? Analyzing CDN logs from ChatGPT, Perplexity, and Claude plus 300K domain studies to verify llms.txt GEO effectiveness.
SPA Sites Visible to Google But Invisible to AI | AI Crawlers Don't Officially Declare JS Rendering Support
Googlebot, Bingbot, and Applebot have evergreen JavaScript rendering. However, GPTBot, ClaudeBot, and PerplexityBot don't officially declare JS execution. We examine whether SPA (CSR) content may be invisible to AI crawlers by reviewing each crawler's official documentation.
How to Check Whether AI Crawlers Are Blocked | GPTBot vs OAI-SearchBot vs ClaudeBot
How to tell training, search, and user-initiated AI crawlers apart — and how to confirm they actually get through, including 403s from WAFs and CDNs outside robots.txt.