Which AI crawlers should I allow in robots.txt?

You opened your robots.txt to add one AI crawler and found a blog post listing forty user-agent strings, half of which you cannot confirm exist. One person on your team wants to block all of them because of the server load. Another read that blocking them removes you from ChatGPT. Both are partly right, and the argument keeps going because every list on the web is organized by company when the actual decision is organized by job.

There are three jobs. Decide each job once and the company-by-company list stops being a decision and becomes a lookup table. This post is that lookup table, plus an honest account of what the file can and cannot enforce.

Every AI crawler is doing one of three jobs

• Retrieval and search. The bot builds or refreshes an index that an assistant consults when it answers a question. This is the bot that decides whether you are eligible to be cited. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, bingbot, Meta-WebIndexer, DuckAssistBot, MistralAI-Index, Amzn-SearchBot.
• Training. The bot collects text that may be used to update model weights. It has nothing to do with whether an assistant can cite you today. GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, Meta-ExternalAgent, CCBot.
• User-triggered fetch. A person asked the assistant to open a specific page, and the assistant goes and gets it live. ChatGPT-User, Claude-User, Perplexity-User, Amzn-User, MistralAI-User, Meta-ExternalFetcher.

The decisions follow directly. Allow retrieval, essentially always — it is the only category where a Disallow line reliably removes you from an answer. Training is a licensing and values call, not a visibility one, and the operators that publish separate tokens say so in their own documentation. User-triggered fetch is barely a decision at all, because several operators state plainly that those fetchers do not honor robots.txt.

Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.

OpenAI, Overview of OpenAI Crawlers

Now set that beside Google's language for Google-Extended, which Google says “does not impact a site's inclusion in Google Search nor is it used as a ranking signal.” Two tokens, two of the largest AI companies, opposite consequences. That asymmetry is why a blanket AI block is a bad instrument.

The complete AI crawler table

Every token below comes from the operator's own documentation. Tokens that circulate in third-party trackers but appear in no operator doc are flagged rather than quietly included.

The awkward cases worth knowing about

Brave is the real exception. Brave deliberately publishes no differentiated crawler user-agent, precisely so sites cannot special-case it, and its documentation states that robots.txt is not used to prevent a page from being indexed. The only route out is a noindex meta directive followed by a re-fetch request at search.brave.com/submit-url. This matters more than it looks: Claude's web search has used Brave as a retrieval backend since March 2025.

xAI publishes nothing. There is no official xAI crawler documentation. Tokens like xAI-Bot and Grok-User appear in third-party trackers with inconsistent claims about robots.txt compliance, none of it confirmed by xAI. You can add those lines; just do not believe they do anything. Whether Grok's web component uses its own crawler or a third-party search API is also unverified.

Bytespider is an observation, not an admission. ByteDance has no published crawler policy page. Bot-management vendors including DataDome classify Bytespider's robots.txt compliance as not respected — third-party monitoring of observed behavior, not something ByteDance has stated. If you care, reach for server-side blocking rather than a Disallow line.

Mistral is an outlier. Mistral's robots documentation lists MistralAI-User and MistralAI-Index and describes both as explicitly not used for generative AI training of any kind. Mistral publishes no training-crawler token at all — it simply has no row in the training column.

There is no Copilot crawler. Microsoft's Copilot Studio documentation confirms that generative answers from public websites are built on Bingbot and Bing Custom Search. Controlling Copilot means controlling bingbot. Any list that gives you a separate Copilot user-agent is inventing one.

`anthropic-ai` is legacy. It appears in a great many copied-and-pasted robots.txt files but is not in Anthropic's current documentation, which lists ClaudeBot, Claude-User and Claude-SearchBot. Leaving the old line in is harmless. Believing it is doing something is not.

What robots.txt cannot do

robots.txt is a request. It has no enforcement mechanism, and the honest version of this post has to say so before it hands you a file to copy.

• A blocked page can still appear. OpenAI's publisher FAQ states that even when a page is blocked, ChatGPT search may still show just the link and page title if the URL reached OpenAI through a third-party provider. Full exclusion requires a noindex meta tag — which means the crawler has to be allowed to fetch the page in order to read the directive telling it to stay out.
• Some fetchers are documented as ignoring it. Perplexity's own docs say Perplexity-User generally ignores robots.txt because a user requested the fetch. Meta documents that Meta-ExternalFetcher and FacebookExternalHit may bypass it. OpenAI says robots.txt may not apply to ChatGPT-User.
• Brave does not use it for indexing at all, as above.
• Non-compliant crawlers ignore it entirely, and you will not know which ones until you read your logs.

A user-agent string is not identification

Anyone can send any user-agent header. Scrapers routinely impersonate well-known bots to inherit their allow rules, and Common Crawl explicitly warns that bots spoof the CCBot user-agent. So the entry in your log that says GPTBot is a claim, not a fact, and if you are making blocking or rate-limiting decisions on the strength of it you are making them on unverified input.

There are two real checks. The first is a forward-confirmed reverse DNS lookup: resolve the requesting IP to a hostname, confirm the hostname belongs to the operator's domain, then resolve that hostname back to an IP and confirm it matches. Common Crawl's crawler resolves to crawl.commoncrawl.org; Google, Bing and Apple document their own equivalents. The second is simpler — several operators publish machine-readable IP ranges.

• OpenAI publishes openai.com/gptbot.json and openai.com/searchbot.json.
• Perplexity publishes perplexity.com/perplexitybot.json.
• Common Crawl documents reverse-DNS verification to crawl.commoncrawl.org.
• Mistral publishes its IP ranges alongside its robots documentation.

Match on IP for anything you enforce at the edge, and treat the user-agent string as a routing hint only. If a bot claiming to be a major crawler is not in the published ranges, it is not that crawler.

Crawl-delay is the middle path

A large share of “block the AI bots” conversations are not about training or citation at all. They are about a small server getting hit harder than it likes. Blocking is a heavy answer to a load problem, and it throws away the citation upside with the traffic.

Anthropic documents support for Crawl-delay, so for ClaudeBot you can slow the crawl instead of killing it. Support is uneven across operators — Google, for one, ignores the directive and expects you to use Search Console's crawl rate settings instead — so treat it as a per-crawler tool rather than a universal one. Where it is honored, it is strictly better than a Disallow for a load complaint.

Two robots.txt files you can copy

Both files maximize your eligibility to be cited and differ only on training. Replace the sitemap URL and sample paths with your own. One thing to know before you paste: robots.txt matching is by most-specific group, so a crawler matching a named group ignores the User-agent: * group entirely. Repeat any site-wide disallows inside the named groups or they will not apply.

# ---- Retrieval and search: allow. These decide whether you can be cited.
User-agent: Googlebot
User-agent: bingbot
User-agent: msnbot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Applebot
User-agent: Amzn-SearchBot
User-agent: DuckAssistBot
User-agent: DuckDuckBot
User-agent: Meta-WebIndexer
User-agent: MistralAI-Index
User-agent: Google-CloudVertexBot
Allow: /
Disallow: /cart/
Disallow: /checkout/

# Note: Applebot and Amazonbot each do retrieval AND training. Apple offers
# Applebot-Extended as a training-only opt-out (below). Amazon publishes no
# equivalent, so Amazonbot stays allowed here — blocking it would also
# remove you from Amazon product search surfaces such as Alexa.
User-agent: Amazonbot
Allow: /

# ---- User-triggered fetch: allow. A human asked for this specific page.
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Amzn-User
User-agent: MistralAI-User
User-agent: Meta-ExternalFetcher
Disallow:

# ---- Model training: disallow.
User-agent: GPTBot
User-agent: Google-Extended
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

# ---- Everything else.
User-agent: *
Allow: /
Disallow: /cart/
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml

Choose this file if your content is your product — a publisher, a research firm, a course business, anyone who may want to license training data later or simply does not want to donate it. You keep every citation path open and give up nothing measurable, because on Google, Anthropic, OpenAI, Apple and Meta the training token is separate from the retrieval token.

# Everything is welcome, including training. No AI-specific exceptions.
User-agent: *
Allow: /
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?*sessionid=

# Optional: rate-limit rather than block where the operator supports it.
# Anthropic documents Crawl-delay support; many others do not.
User-agent: ClaudeBot
Crawl-delay: 10

Sitemap: https://example.com/sitemap.xml

Choose this file if your content is marketing for something else you sell — most SaaS, agencies, local services, ecommerce. Your pages are not the asset; the business behind them is. Every extra place your name and product language can appear is upside, and you never have to revisit the file when a new token appears.

If you cannot decide, ship the second file. A permissive robots.txt is the lower-regret default, because a training opt-out is reversible in an afternoon and a citation you never earned is not recoverable at all.

The checklist

1. Fetch your current robots.txt and read it end to end. Note every Disallow that hits a retrieval crawler — those are the lines actually costing you.
2. Confirm Googlebot, bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot and DuckAssistBot are all allowed. This is the citation-eligibility set.
3. Make the training decision once, as a business decision, and apply it to GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, Meta-ExternalAgent and CCBot together.
4. Leave the user-triggered fetchers allowed. Blocking them mostly annoys people who explicitly asked for your page, and several operators do not honor the block anyway.
5. Add a Sitemap: line if one is missing. It is the cheapest thing in the file.
6. For anything you genuinely need out of an index, use noindex — and allow the crawler to fetch the page so it can read the directive.
7. Pull a week of server logs, verify the top bots by IP rather than user-agent, and apply Crawl-delay where load is the real complaint.
8. Re-check the operator docs every six months. Four of the tokens in the table above did not exist eighteen months ago.

That is the whole job. robots.txt is not a growth lever — a correct file prevents self-inflicted damage rather than creating visibility. The thing worth checking next is whether the pages those crawlers are now allowed to read actually answer the questions people ask, which is where a tool like GetFound3 earns its keep.