AI crawler access
what each bot does, and what blocking it costs

Which AI crawlers actually exist?

Nine documented agents across three vendors, plus Google's training control. Every one of them is published by the vendor, and the descriptions below are the vendors' own wording rather than our interpretation.

Two of those rows are worth reading twice. Perplexity-User does not honour robots.txt by Perplexity's own documentation, so if you need it stopped, that has to happen at your firewall or CDN and not in a text file. And Google-Extended does not gate AI Overviews or AI Mode. Allowing it buys you no AI Overviews visibility, and anyone who told you otherwise has not read Google's crawler documentation.

Does blocking GPTBot stop ChatGPT from citing you?

No. GPTBot is the training crawler. ChatGPT search visibility runs through OAI-SearchBot and ChatGPT-User, which are separate agents with separate directives.

This distinction is where most of the accidental damage in this field happens. A publisher decides they do not want their content used to train a commercial model, adds a disallow rule for GPTBot to robots.txt, and is satisfied. That is a coherent position and it does what they intended. The problem starts when a wildcard rule, a template from a blog post, or a security plugin blocks anything with the string GPT or OpenAI in it, which takes OAI-SearchBot and ChatGPT-User down with it. The publisher has now removed themselves from ChatGPT search while believing they only opted out of training.

The same split applies at Anthropic, where ClaudeBot is training and Claude-User and Claude-SearchBot are search, and at Perplexity, where PerplexityBot is explicitly documented as not being used for foundation model training at all.

Decide the two questions separately and write them separately.

• Do you want your content in model training corpora? Reasonable people answer this differently and it is genuinely your call.
• Do you want to appear in AI assistants' answers? For almost every B2B company selling something, the answer is yes.

What should a B2B site put in robots.txt?

For a company that wants to be found and cited, allow everything and be explicit about it. Silence works, since absence of a directive means allowed, but explicit entries are self documenting and they survive the next person who edits the file.

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: CCBot
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Claude-SearchBot is the one most allowlists published in 2025 are missing, because it was documented later than the other two Anthropic agents and it is the one that builds Claude's search index. If you copied your list from a post written last year, check for it.

Why is your site blocked when robots.txt says allow?

Because the block is above your application. Cloudflare began blocking AI crawlers by default for new domains on 1 July 2025, asking new domain owners up front whether to permit AI crawling, and it sits in front of a large share of the web. A domain configured by a developer who clicked through that dialogue in 2025 can be serving a challenge page to every AI crawler while its robots.txt politely says everything is allowed.

The general form of this failure matters more than the specific vendor. Your bot policy is enforced at the outermost layer that inspects the request, which is usually your CDN or WAF, and robots.txt is a request that a well behaved crawler makes after it has already got past that layer. If the edge returns 403, the file is never read.

curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://yourdomain.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "PerplexityBot" https://yourdomain.com/

Anything other than 200 is a problem you cannot fix in robots.txt. Also check for a soft block, which is a 200 response containing a challenge or interstitial instead of your page. That one passes the status code check and fails everything that matters, so read the body as well as the code.

What does allowing AI crawlers actually cost you?

Three real costs and one imagined one, and the imagined one is the reason most of these conversations go badly.

Real cost one, training inclusion. If GPTBot and ClaudeBot are allowed, your content contributes to model training and you receive nothing directly for it. If your business is publishing, this is a genuine commercial decision with real revenue attached and we are not going to pretend it is obvious. If your business is selling B2B services, your content was written to persuade buyers and a model that has absorbed it is a model that can describe you accurately.

Real cost two, bandwidth. Crawler traffic is not free. On a large estate it is a line item worth measuring rather than assuming.

Real cost three, losing control of framing. A model that has read your site will describe your company in its own words. Sometimes wrongly. The response to that is entity correction work, not a block, because blocking does not remove existing descriptions, it removes your ability to influence them.

The imagined cost, traffic cannibalisation. The fear that being quoted means never being visited. It is real for informational queries with a one line answer and it is much weaker for considered B2B purchases, where the assistant's answer is the start of the evaluation and not the end of it. Being the source the assistant names is a strong position. Being absent from the answer entirely does not protect your traffic. It loses you the buyer earlier in the process, at the stage where you cannot see it happening.

What do we not know about crawler access?

Three honest gaps, stated because a page that only tells you what it knows is less useful than one that marks the edges.

We do not know how OpenAI ranks candidates for citation once it has fetched them. It is undocumented, and the two public studies attempting to infer it from Bing rankings disagree by an order of magnitude, which means at least one of them is wrong and possibly both.

We do not know whether ChatGPT operates its own index alongside Bing. The evidence is circumstantial behaviour testing and nothing more.

We do not know whether Anthropic uses Brave as its search provider. The circumstantial case is strong, Brave appears on Anthropic's published subprocessor list and a Brave parameter is exposed in Anthropic's web search API, but Anthropic has never said so and we are not going to state it as fact.

If you are choosing between advisors in this field, ask them these three questions. The answer you want is the one that says nobody knows.