On October 30, 2026, Google, Microsoft Bing, OpenAI, and Anthropic each said they will honor one shared robots.txt token for AI training opt-outs, __usage_ai-training. Bing is also splitting grounding into its own page-level control. The change punishes the blanket block, which one vendor-benchmark audit puts on 41% of B2B sites. Here is what each switch controls and what to fix before enforcement starts on February 1, 2027.
What changed on October 30
One line in robots.txt now speaks to four AI companies at once. Internet-Draft draft-kelley-aipref-training-preference-signal-00, written by Kevin Kelley of Anthropic and published October 30, defines a "policy token", __usage_ai-training. A site writes an ordinary robots.txt group under that token, and participating providers read it as the site's preference about use of its content for generative AI training. All four companies endorsed it the same day.
The draft's own example opts the whole site out of training except one folder:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/The mechanism adds no new syntax. Path rules under the token follow RFC 9309, the robots.txt standard, including longest-match precedence, and parsers that do not know the token ignore the group like any unknown user agent. The draft is an individual submission with no formal IETF standing; it expires May 3, 2027 unless revised. Support is announced, not live, and each company gave a different date:
| Provider | What they announced | When it takes effect |
|---|---|---|
| Microsoft Bing | Shared token for training, plus new page-level ai-training and ai-grounding controls. NoArchive and NoCache retire. |
February 1, 2027; old tags stop counting March 1, 2027 (official) |
| Shared token. When Google-Extended is not specified, a token opt-out is read as a Google-Extended disallow. | 2027 target; crawler docs say early 2027 (official) | |
| OpenAI | Shared signal. Existing GPTBot and OAI-SearchBot controls stay in place. | "In the coming months" (official) |
| Anthropic | Draft author. Implementation has started across its services. | In progress (official) |
Meta is named in the draft's acknowledgments and publicly welcomed the signal, but has not committed to honor it (The Verge). The list of companies that actually obey the token is four today. Treat it as a preference those four have promised to read, not a law of the web.
Training, grounding, and crawling are three different switches
The token says how content may be used, not whether a crawler may fetch it. "This policy token is not a directive for crawling behavior," the draft states. Three separate switches are now in play: crawl access (unchanged, per-crawler rules); training use (the new shared token); and grounding use, which feeds AI answers (page-level controls at Microsoft; Google-Extended at Google).
Four limits in the draft matter more than the syntax:
- Wildcard rules do not count. "Wildcard groups (i.e., User-agent: *) MUST NOT be interpreted as matching the policy token." A blanket
Disallow: /under*blocks crawlers but expresses no training preference. - No group means no preference. A file without the token "expresses no preference at all". Silence is not a no.
- Private agreements override. "Private agreements MAY override preferences expressed via this mechanism."
- Specific beats general. Consumers may treat proprietary tokens like Google-Extended or GPTBot as more specific than the shared one.
Google routes the shared token to the Google-Extended switch. Its Search Central announcement says that when a site opts out with the token and does not mention Google-Extended, Google plans to read that "as if the publisher had disallowed Google-Extended" (official). Google-Extended governs training of Gemini models and grounding in Gemini Apps and Vertex AI, and it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" (official docs). The same docs note that sites which specifically allow Google-Extended are eligible to appear in Gemini Apps answers and receive clicks. Put those two official statements together and our reading, not a Google sentence, is this: a blanket training opt-out left alone also takes a site out of grounding in Gemini Apps. If you want those citations, allow Google-Extended explicitly.
Bing went further and made grounding its own page-level setting, carried by a meta tag or an X-Robots-Tag header (official):
<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallow">
<meta name="Bingbot" content="ai-grounding:disallowextended">There is no site-wide robots.txt grounding preference; grounding is page-level only. DisallowExtended keeps content eligible for Microsoft-operated experiences such as Copilot, AI Mode, and Microsoft 365 while withholding it from grounding services Microsoft sells to third parties, like Web IQ. Both preferences default to Allow when nothing is published. Microsoft's own caution: selecting Disallow or DisallowExtended will significantly reduce your visibility in AI-generated experiences, though neither changes eligibility in regular Bing results.
The blanket block is the expensive mistake
Blocking "the AI bots" as one group was always a category error. Training crawlers and answer crawlers are different user agents, and blocking the second kind removes a site from the answers buyers see. The new controls make the distinction impossible to avoid.
The market already separates the two, unevenly. Cloudflare Radar's September 28, 2026 snapshot of 4,242 robots.txt files shows every major training crawler disallowed more often than allowed: Bytespider 3.83:1, CCBot 2.61:1, GPTBot 1.60:1, ClaudeBot 1.58:1, Google-Extended 1.38:1 (Cloudflare Radar data via TechnologyChecker). The answering crawlers flip the other way: OAI-SearchBot 0.91:1, bingbot 0.99:1. An independent crawl of 5,467 robots.txt files found 19.9% of sites blocking at least one training crawler at the root against 11.4% blocking an AI search crawler, and 454 of the 874 GPTBot blockers still allowing OAI-SearchBot (Citevisor, October 2026). Operators who understood the split kept the right door open.
The ones who did not are the cautionary number. CapstonAI's Q1 2026 cohort audit (a vendor benchmark, drawn from its own client base) puts 41% of B2B sites blocking at least one major AI bot, usually a leftover from the 2023-2024 "block everything" panic (CapstonAI). Each of those blocks was written when the only switch available was the crawl-level kill switch. From February 1, 2027 the switches separate, and a site that still blocks at the wrong layer quietly disappears from answers it never meant to leave. That matters because the answers are where buyers now start; our earlier note covers why being named by AI beats ranking on Google.
What to block and what to leave open
Decide per content type, not per company. Training is a rights decision you can now express once, path by path. Answer crawlers and user-triggered fetchers are distribution: leave them open for anything you want cited.
| Content | Training (the new token) | Answer and user-fetch crawlers |
|---|---|---|
| Public pages you want cited | State your preference with __usage_ai-training, path by path. |
Leave open. Blocking these removes you from the answers buyers see. |
| Paywalled, licensed, or premium content | Disallow, via the token group or per-bot rules. | Your call. Grounding is Bing's page-level ai-grounding setting. |
| Pages you want used (press kit, open data) | Allow. This is the draft's own example. | Leave open. |
| Staging, admin, internal paths | Block crawling with product tokens or *, as today. |
Same. The token manages use, not access. |
One row deserves a caveat. User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) fire when a person asks about you inside a chat. OpenAI documents robots.txt as possibly not applying to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it (Citevisor). For that cluster, robots.txt is a weak instrument.
What to do before February 1, 2027
Nothing is enforced yet, which is exactly why the next three months are worth using. Fixes only take effect after engines recrawl a site, and Microsoft warns that publishing a setting is not the same as activating it. The sequence:
- Read your robots.txt today. List every AI token in it and sort them into training, answering, and user-fetch. Our free instant check does the first pass in seconds.
- Retire stale blanket blocks. Anything added in 2023-2024 "to be safe" deserves a fresh decision against the table above.
- If you publish NoArchive or NoCache, act before March 1, 2027: publish the new preferences and record the temporary grounding preference in Bing Webmaster Tools, which opens December 15, 2026. Miss the date and those pages default to Allow for grounding until recrawl.
- If you want Gemini Apps citations, allow Google-Extended explicitly. Once you adopt the token, silence reads as disallow.
- Add the token group deliberately, path by path, using the draft's example as the pattern.
- Verify and re-check monthly. The manual method takes an afternoon and no tools.
The window before enforcement
As of November 4, 2026, none of the four companies honors the token yet, and adoption is near zero. joinmassive checked 49 news sites on November 1 and found zero deployments of the token; 10 of the 33 homepages it could load still carry NoArchive or NoCache tags (joinmassive).
Where we fit
We build visibility diagnostics for a living, and crawler access sits at the core of them. CiteGap's free instant check reads your robots.txt and reports which AI crawlers are blocked: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended. It takes seconds. For the full picture of what AI says about your product, the written diagnosis runs 30 buyer questions and hands you a fix list.
Frequently asked questions
What is the __usage_ai-training token in robots.txt?
A shared "policy token" defined in IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00. A site writes an ordinary robots.txt group under User-Agent: __usage_ai-training, and the Disallow and Allow rules under it express the site's preference about use of each path for generative AI training. Google, Microsoft Bing, OpenAI, and Anthropic announced on October 30, 2026 that they will honor it, with start dates from early 2027 onward depending on the company.
What is the ai-grounding token?
Microsoft's page-level AI Grounding preference. Set it with <meta name="Bingbot" content="ai-grounding:disallow"> (or the X-Robots-Tag header) to keep a page out of AI-generated answers, or ai-grounding:disallowextended to allow Microsoft-operated experiences like Copilot while withholding third-party grounding services. There is no site-wide robots.txt grounding preference, and grounding defaults to Allow when nothing is published.
Does User-agent: * Disallow: / opt my site out of AI training?
No. The draft says wildcard groups MUST NOT be interpreted as matching the policy token, and a robots.txt file with no __usage_ai-training group expresses no preference at all. The blanket block still governs crawling, which is a separate switch from use.
Will an AI training opt-out hurt my Google Search rankings?
No. Google's crawler documentation states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal there (official), and the shared token is read through the Google-Extended control. Microsoft likewise says its new preferences do not change eligibility in regular Bing search results.
Should I block AI crawlers in robots.txt?
Separate the switches. Block crawling of genuinely sensitive paths as you always did. Express your training preference with the shared token, path by path. Leave answer crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, bingbot) and user-triggered fetchers open for anything you want cited, because blocking those removes your pages from the answers buyers see.
When do the AI companies start honoring the token?
Microsoft Bing starts February 1, 2027 (official). Google targets 2027, its crawler docs say early 2027. OpenAI says "in the coming months" (official). Anthropic has started implementing. As of November 4, 2026, no company honors the token yet.
Sources
- Block-to-allow ratios, training crawler Bytespider 3.83:1 vs answer crawler OAI-SearchBot 0.91:1 (Cloudflare Radar, September 28, 2026 snapshot, 4,242 robots.txt files; GPTBot 1.60:1, ClaudeBot 1.58:1, Google-Extended 1.38:1, CCBot 2.61:1, bingbot 0.99:1): TechnologyChecker analysis of Cloudflare Radar data
- 19.9% of sites block at least one AI training crawler at the root vs 11.4% blocking an AI search crawler; 454 of 874 GPTBot blockers allow OAI-SearchBot (independent crawl, 5,467 robots.txt files, October 2026): Citevisor 2026 robots.txt study
- 41% of B2B sites block at least one major AI bot (Q1 2026 cohort audit; vendor benchmark, CapstonAI's own client base): CapstonAI robots.txt guide
- IETF Internet-Draft draft-kelley-aipref-training-preference-signal-00, "Machine-Readable Training Preference Signal" (Kevin Kelley, Anthropic, October 30, 2026; token semantics, crawl-independence, wildcard and precedence rules): datatracker.ietf.org
- Kelley's announcement post to the IETF working group list (Anthropic implementation status; "expresses no preference at all"): mailarchive.ietf.org
- Google Search Central Blog, "Supporting a shared AI training control for websites" (John Mueller, October 30, 2026; Google-Extended interpretation, Gemini Apps grounding eligibility, 2027 target): developers.google.com
- Google crawler documentation, Google-Extended token (no effect on Search inclusion or ranking; grounding scope; early 2027 support note): developers.google.com
- Microsoft Bing Blog, "Your content, your choice: clearer preferences for training and grounding" (Krishna Madhavan, October 30, 2026; page-level ai-training / ai-grounding controls, DisallowExtended, timeline Feb 1 / Mar 1 / Sep 30 2027, NoArchive and NoCache retirement): blogs.bing.com
- OpenAI, "Making it easier for websites to express AI training preferences" (October 30, 2026; "coming months"; existing crawler controls continue): openai.com
- The Verge, coverage of the four-company adoption and Meta's non-commitment (October 30, 2026): theverge.com
- joinmassive, deployment check of 49 news sites on November 1, 2026 (0 tokens deployed; 10 of 33 loadable homepages still carry NoArchive/NoCache): joinmassive.com