How to Configure AI Crawler Control in robots.txt in 2026

Q: Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot is the training crawler only. The bot that builds the index behind ChatGPT search answers is OAI-SearchBot , and the one that fetches a page live when someone asks about it is ChatGPT-User . OpenAI documents that each setting is independent, and that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, so blocking GPTBot alone leaves your citations intact.

Q: Does Google-Extended control whether I appear in AI Overviews?

It does not. Google documents Google-Extended as controlling whether crawled content may be used for training future Gemini models and for grounding in the Gemini apps and Vertex AI. AI Overviews and AI Mode are part of Google Search, so they follow the Search index and the snippet controls: nosnippet , data-nosnippet , max-snippet and noindex . Disallowing Google-Extended will not take you out of AI Overviews.

Every major AI vendor now ships separate robots.txt tokens for model training, for search indexing, and for live user-triggered fetches, and blocking the wrong one quietly removes you from AI answers. This page gives you the verified token list and three complete rule sets you can paste into /robots.txt today.

Check Your Site Now Browse the Glossary

CategoryCrawler access, licensing & indexing StatusStable Maintained byPer-vendor conventions Glippy checkMachine Readability (category 6)

AI Crawler Control in robots.txt (GPTBot, ClaudeBot, Google-Extended and friends) is the per-bot use of User-agent groups to give each AI vendor's crawler its own answer about what it may fetch. The tokens split three ways: training crawlers that harvest text for model training, search crawlers that build the index an assistant cites from, and user-triggered fetchers that pull one page live because a person asked a question. Blocking a training token costs you nothing in AI answers, while blocking a search token removes you from them.

Why AI Crawler Control in robots.txt matters for AI visibility

OpenAI, Anthropic, Perplexity, Apple, Google and Amazon each run more than one bot, and the difference between them is the whole point. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and that each of its bot settings is independent of the others, so you can allow the search crawler while refusing the training crawler. A blanket AI block copied off a forum usually hits both, which trades an opt-out you wanted for an absence you did not. For the background on who runs these bots and how the vendor politics got here, the conceptual explanation lives here. The rest of this page is the file itself.

The second failure is the grammar of robots.txt rather than the token list. Under RFC 9309 a crawler obeys exactly one group: the group naming its product token, falling back to User-agent: * only when no group names it. So adding a two-line courtesy group for GPTBot deletes every wildcard Disallow you had written for it. Google documents the same behaviour, stating that only one group is valid for a particular crawler and that other groups are ignored. Every example below repeats its rules inside each named group for exactly that reason.

Where the spec lives

Every token on this page was checked against the operator's own bot documentation in August 2026. Do not lift a token from a blog post: vendors rename them, and a misspelled token matches nothing at all, which reads as allow.

RFC 9309: Robots Exclusion Protocol - the normative grammar. Read it for group precedence, the longest-match rule, the 500 KiB parse limit, and what a 5xx on robots.txt means.
OpenAI: bots and crawlers - the current tokens GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot, with the published IP JSON files for verification.
Anthropic: web crawling and how to block it - ClaudeBot, Claude-User and Claude-SearchBot, plus the only major vendor that documents Crawl-delay support.
Google: common crawlers - the authoritative wording for Googlebot, GoogleOther, Google-CloudVertexBot and Google-Extended. Check here before assuming what Google-Extended covers.
Apple: about Applebot - the Applebot and Applebot-Extended split, and the statement that pages disallowing Applebot-Extended can still appear in search.
Perplexity: bots - PerplexityBot and Perplexity-User, including the note that the user-triggered agent generally ignores robots.txt.

Three ways to implement AI Crawler Control in robots.txt

The three postures below cover most real sites. The first is a static robots.txt for publishers who want citations but not training, which is where most commercial sites land. The second is for sites that want every bot in and only need the traffic to behave, so the enforcement moves to the server where it is actually binding. The third generates the file from application code, which is the honest option once your access rules follow your route structure rather than a fixed list.

Open to AI search and answers, closed to model training

Use this when you want to be cited by ChatGPT, Claude, Perplexity, Google and Siri, but do not want your text in a training corpus. It is a static file at the domain root. Note the repeated Disallow lines: each named group is self-contained, so a bot named here never sees the wildcard group.

text/robots.txt

# Search and answer engines. These are the bots that cite you.
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Amzn-SearchBot
Allow: /
Disallow: /cart/
Disallow: /checkout/

# Live fetchers. A person asked for this page just now.
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Amzn-User
Allow: /
Disallow: /cart/
Disallow: /checkout/

# Training crawlers. No.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /
Disallow: /cart/
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml

What this does: Every crawler that can put a link to you in an AI answer keeps full access, while the eight training tokens get a flat refusal. Two costs are worth knowing before you ship it: Google-Extended covers grounding in the Gemini apps and Vertex AI as well as training, so this file gives up those citations, and blocking Amazonbot is what stops Amazon training while Amzn-SearchBot keeps you eligible for Alexa and Rufus.

Fully open, with the rate limit enforced at the server

Use this when you want every AI system to have your content and the only real problem is a crawler hammering an expensive origin. Crawl-delay is not a reliable answer: Google documents that it is unsupported, and Amazon documents that none of its three agents support it. Serve an open robots.txt and put the limit somewhere binding.

nginx/etc/nginx/conf.d/ai-crawlers.conf

# Classify AI bots by user agent. An empty key is never rate limited,
# so ordinary browser traffic passes through untouched.
map $http_user_agent $ai_bot {
    default                "";
    "~*GPTBot"             openai;
    "~*OAI-SearchBot"      openai;
    "~*ClaudeBot"          anthropic;
    "~*Claude-SearchBot"   anthropic;
    "~*PerplexityBot"      perplexity;
    "~*Amazonbot"          amazon;
    "~*Meta-ExternalAgent" meta;
    "~*CCBot"              commoncrawl;
}

limit_req_zone $ai_bot zone=aibots:10m rate=2r/s;

server {
    listen 443 ssl;
    server_name example.com;

    location = /robots.txt {
        add_header Content-Type text/plain;
        return 200 "User-agent: *\nAllow: /\nDisallow: /api/\n\nSitemap: https://example.com/sitemap.xml\n";
    }

    location / {
        limit_req zone=aibots burst=20 nodelay;
        limit_req_status 429;
        try_files $uri $uri/ /index.html;
    }
}

What this does: robots.txt says yes to everyone, and each vendor gets its own 2 requests per second bucket with a burst of 20 before nginx returns 429. Because requests with an empty key are not accounted, human visitors are unaffected. A well-behaved crawler backs off on 429; one that does not is worth checking against the vendor IP list before you assume it is genuine.

Selective by path, generated from the app router

Use this when the answer depends on the route rather than the bot: public docs open to everyone, gated reports open only as a teaser, internal routes closed to all. Generating the file keeps it in step with the routes instead of drifting from them. This is the Next.js app router convention, available since v13.3.

typescriptapp/robots.ts

import type { MetadataRoute } from 'next'

const HOST = 'https://example.com'

// Never served to any bot.
const PRIVATE = ['/account/', '/admin/', '/api/', '/preview/']
// Gated: the summary route is public, the full report is not.
const GATED = ['/members/', '/reports/']
const TEASER = '/reports/summary/'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      // Each object becomes its own complete group. A bot named here
      // ignores the '*' group, so repeat every rule it still needs.
      {
        userAgent: ['Googlebot', 'Bingbot', 'Applebot', 'OAI-SearchBot',
                    'Claude-SearchBot', 'PerplexityBot'],
        allow: ['/', TEASER],
        disallow: [...PRIVATE, ...GATED],
      },
      {
        userAgent: ['ChatGPT-User', 'Claude-User', 'Perplexity-User'],
        allow: ['/', TEASER],
        disallow: [...PRIVATE, ...GATED],
      },
      {
        userAgent: ['GPTBot', 'ClaudeBot', 'Google-Extended',
                    'Applebot-Extended', 'Amazonbot', 'CCBot'],
        disallow: '/',
      },
      { userAgent: '*', allow: '/', disallow: [...PRIVATE, ...GATED] },
    ],
    sitemap: `${HOST}/sitemap.xml`,
    host: HOST,
  }
}

What this does: Next.js emits one User-Agent group per name in each array, so the output is a valid multi-group file rather than a merged one. The teaser works because of the longest-match rule: for /reports/summary/q3 the 17-octet Allow: /reports/summary/ beats the 9-octet Disallow: /reports/, and RFC 9309 says an allow wins an exact tie.

Implementation guidelines

These are the things that actually break in production, in the order they tend to bite.

Repeat every rule inside every named group. A crawler uses the group naming it and ignores User-agent: * entirely. Adding a bare User-agent: GPTBot plus Crawl-delay: 1 silently grants GPTBot access to everything your wildcard group was fencing off.
Do not expect robots.txt to stop a user-triggered fetcher. OpenAI says robots.txt rules may not apply to ChatGPT-User because the action is user-initiated, Perplexity says Perplexity-User generally ignores robots.txt, Meta-ExternalFetcher may bypass it, and Google's user-triggered fetchers ignore it. Anthropic is the outlier and documents Claude-User as honouring it. If you need a hard stop, enforce it at the edge.
Serve robots.txt as a 200 and keep it small. RFC 9309 requires crawlers to assume complete disallow on a 5xx, so a CDN error page on /robots.txt is a site-wide block. Both the RFC and Google cap parsing at 500 KiB, and anything after that is ignored.
Know what each token does not cover. Google-Extended governs training and grounding in the Gemini apps and Vertex AI, not AI Overviews or AI Mode, which follow the Search index plus nosnippet, data-nosnippet, max-snippet and noindex. Applebot-Extended governs training but not Siri or Spotlight. Microsoft publishes no training token at all and uses the noarchive and nocache meta values instead.
Treat the token as a claim, not an identity. Anyone can send a user agent string. OpenAI publishes IP ranges at openai.com/gptbot.json and its siblings, Anthropic at claude.com/crawling/bots.json, Common Crawl at index.commoncrawl.org/ccbot.json, and Google and Bing support reverse DNS. Verify before you build access logic on the name.
Check the edge as well as the file. Bot managers block matching requests before they reach your origin, so an Allow: / that returns 403 to the bot is still a block. From 15 September 2026, newly onboarded Cloudflare domains block Training and Agent bots by default on pages that display ads, with Search left allowed.
Re-verify the token list on a schedule. Tokens get added, renamed and deprecated: Google-NotebookLM became Google-GeminiNotebook, Amazon split one bot into Amazonbot, Amzn-SearchBot and Amzn-User. A quarterly diff against the vendor bot pages is cheap; discovering a stale block six months later is not.

Do this, not that

Do

Repeat every Disallow inside each named group, because a named group replaces the wildcard group rather than extending it.
Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Applebot if you want to be cited at all.
Block Amazonbot but keep Amzn-SearchBot when the goal is no training with continued Alexa and Rufus coverage.
Test a real request with the bot's user agent string against your live origin, not just against the text of the file.

Do not

Do not block Googlebot to stay out of AI Overviews. That removes you from Search as well; use nosnippet or max-snippet instead.
Do not add a Crawl-delay line for a bot without repeating that bot's Disallow rules in the same group.
Do not paste a long allowlist of tokens you have not verified. A misspelled token matches no crawler and therefore blocks nothing.
Do not put noindex in robots.txt. Google does not support it as a robots.txt field; it belongs in a meta robots tag or an X-Robots-Tag header.

How Glippy checks this

Glippy fetches /robots.txt, parses the per-user-agent groups, and then classifies every matched bot against its own registry of training crawlers versus citation crawlers. Blocking a training crawler is reported as informational and barely moves the score, because opting out of training is a legitimate choice. Blocking a citation crawler is a warning at one or two bots and a failure beyond that, since it directly removes you from answer engines. Glippy is deliberately conservative about the wildcard group: it treats a bot as blocked when either its own group or User-agent: * disallows the root, unless its own group carries an explicit root Allow. It also picks up Sitemap references and Content-Signal directives from the same file. The result scores under Machine Readability (category 6), and you can run it on its own with the AI crawlability checker.

Check your AI Crawler Control in robots.txt setup

Glippy runs 240+ checks across 16 categories on any page, including Machine Readability (category 6). No sign-up required.

Try the Glippy Chrome extension

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot is the training crawler only. The bot that builds the index behind ChatGPT search answers is OAI-SearchBot, and the one that fetches a page live when someone asks about it is ChatGPT-User. OpenAI documents that each setting is independent, and that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, so blocking GPTBot alone leaves your citations intact.

Does Google-Extended control whether I appear in AI Overviews?

It does not. Google documents Google-Extended as controlling whether crawled content may be used for training future Gemini models and for grounding in the Gemini apps and Vertex AI. AI Overviews and AI Mode are part of Google Search, so they follow the Search index and the snippet controls: nosnippet, data-nosnippet, max-snippet and noindex. Disallowing Google-Extended will not take you out of AI Overviews.

Why did adding a Crawl-delay line for one bot make it ignore my other rules?

Because you created a group that names that bot. Under RFC 9309 a crawler obeys the group matching its product token and falls back to User-agent: * only when no group names it, and Google states the same rule as only one group being valid per crawler. Groups are not layered, so the bot stopped inheriting the wildcard rules. Copy every Disallow line into the new group.

Can robots.txt stop an AI agent that a user sent to my site?

Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because the request is user-initiated, Perplexity says Perplexity-User generally ignores robots.txt, Meta says Meta-ExternalFetcher may bypass it, and Google states its user-triggered fetchers generally ignore robots.txt. Anthropic is the exception and documents Claude-User as honouring robots.txt. Treat robots.txt as a stated preference for these agents and enforce anything mandatory at the CDN or origin.

Reviewed against the primary sources on 27 August 2026. These standards move quickly, so check the linked specs before you ship.