Home › AI Search › AI crawler management
AI Crawlers: Which to Allow, Which to Block, and Why
Blocking AI crawlers and staying visible in AI search are two separate decisions, and treating them as one is the most expensive mistake we see. Training crawlers such as GPTBot and ClaudeBot decide whether your content may help improve a model. Search crawlers such as OAI-SearchBot and PerplexityBot decide whether you can be found and quoted when somebody asks a question today. Block the second group and you disappear from the answer. Block the first and, in most cases, you do not.
The two kinds of crawler, and why the distinction is the whole point
Every major AI provider now runs more than one crawler, and publishes what each is for. They are not interchangeable, and a single blanket rule in your robots.txt will usually catch both kinds.
OpenAI's documentation describes GPTBot as crawling content that may be used to train their foundation models, and OAI-SearchBot as being used to surface websites in ChatGPT's search features. Anthropic distinguishes ClaudeBot, which collects content that could contribute to training, from Claude-User, which fetches a page because somebody asked Claude a question, and Claude-SearchBot, which works on search quality. Perplexity states plainly that PerplexityBot surfaces and links sites in results and is not used to crawl content for foundation models.
So the question is never simply whether to block AI. It is which of your two quite different objections you are acting on.
The crawlers that matter, and what each one controls
| Crawler | Run by | What it controls | Our default |
|---|---|---|---|
| Googlebot | Google Search, including AI Overviews and AI Mode | Allow | |
| Google-Extended | Whether content trains and grounds Gemini models. Does not affect Search inclusion or ranking | Your call | |
| Bingbot | Microsoft | The Bing index, which also serves Copilot | Allow |
| GPTBot | OpenAI | Training OpenAI foundation models | Your call |
| OAI-SearchBot | OpenAI | Being surfaced in ChatGPT search | Allow |
| ChatGPT-User | OpenAI | Fetching a page because a user asked | Allow |
| ClaudeBot | Anthropic | Training Anthropic models | Your call |
| Claude-User | Anthropic | Fetching a page for a user question | Allow |
| Claude-SearchBot | Anthropic | Search result quality | Allow |
| PerplexityBot | Perplexity | Being surfaced and linked in Perplexity | Allow |
| Perplexity-User | Perplexity | Fetching a page for a user question | Allow |
One entry on that table is worth reading twice. Google's own documentation states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal in Google Search. It governs Gemini training and grounding. It is not the AI Overviews switch, whatever you have been told.
The AI Overviews control is somewhere else entirely
Since June 2026 there has been a separate control in Search Console that prevents a site's content appearing in Google's generative AI features, including being linked within them. Google is careful to say that using it carries no penalty elsewhere in Search. Alongside it sits a generative AI performance report, which gives you impressions from AI Overviews and AI Mode as a reportable number for the first time.
If someone has advised you to set Google-Extended in order to control AI Overviews, they have conflated two different settings. We have now seen that in three separate audits this year.
The case for blocking, made fairly
There are honest reasons to block training crawlers. If your content is your product, as it is for a publisher, a course provider or a research firm, letting it train a model that then answers the question for free is a real commercial loss. Some organisations also have licensing obligations that make it simpler to opt out. Those are legitimate positions and we would not argue you out of either.
What is not a good reason is a vague sense that AI is taking something. Blocking a training crawler does not remove content already learned, does not stop a user pasting your page into a chat window, and does very little to protect anything. It is a signal of intent more than a lock.
What blocking does not do
- It does not remove you from what a model already learned. Training happened before your rule existed.
- It does not stop a person copying your page into an assistant themselves.
- It does not affect Google Search. Only a robots rule aimed at Googlebot does that, and you should not write one.
- It is not enforcement. robots.txt is a request that well behaved crawlers honour, not a technical barrier.
A sensible starting point
Allow everything that decides whether you can be found. Make a deliberate decision about training. If you are a normal business selling a service rather than selling content, the balance is usually to allow both, because the cost of being invisible is real and the cost of contributing to a model is abstract.
Whatever you choose, write a comment in the file explaining the decision and the date. The next person to open your robots.txt, who may be you in eighteen months, will need to know whether a rule was a considered choice or something a plugin added.
A warning about AI visibility checking tools
If a tool reports that your site blocks AI crawlers, verify it by reading your own robots.txt before you believe it. We ran an automated pass across nineteen sites earlier this year and two of them scored between 25 and 30 on the first attempt and 85 and 90 on a retry, because bot protection had served an interstitial that is indistinguishable from an empty page. A site turning away an automated request from a data centre is not evidence that it blocks anybody.
It is also worth knowing that AI crawlers do not behave like Googlebot. As Jono Alderson has pointed out, these systems do not crawl the web the way Google does and do not necessarily render your pages or execute your scripts. Content that only appears after JavaScript runs may simply not be seen.
Read next
Frequently asked questions
Will blocking GPTBot stop ChatGPT recommending my business?
Not on its own. GPTBot governs training. Surfacing a site in ChatGPT's search features is OAI-SearchBot, which is a separate token. If you block both, which a single blanket rule often does, then yes, you have removed yourself from the answer.
Does robots.txt actually stop anybody?
It is a request, not a barrier. Google, OpenAI, Anthropic and Perplexity all state that their crawlers respect it, and in our experience they do. It has no effect at all on a person copying your page into a chat window themselves, or on crawlers that ignore conventions.
What is CCBot and should I worry about it?
CCBot belongs to Common Crawl, a non profit that publishes an open web dataset which many organisations, including AI developers, have used. Blocking it is a reasonable choice if you are blocking training generally. It has no bearing on search visibility.
If a checking tool says my site is unreachable, does that mean I am blocking AI?
No, and this is worth being firm about. Bot protection and rate limiting routinely serve an interstitial or an empty response to automated requests, which looks identical to a blocked site. Read your own robots.txt, and retry from a different network, before acting on it.
Not sure how AI assistants describe your business?
We run a hand checked AI visibility audit, scored out of 100 across five areas, with the weightings published so you can argue with them.
Book a 15 minute callSources
- OpenAI. Overview of OpenAI crawlers. Checked 20 August 2026.GPTBot for training, OAI-SearchBot for ChatGPT search, ChatGPT-User for user actions, OAI-AdsBot for ad safety.
- Anthropic. Does Anthropic crawl data from the web, and how can site owners block the crawler?. Updated 7 April 2026.ClaudeBot for training, Claude-User for user questions, Claude-SearchBot for search quality.
- Perplexity. Perplexity crawlers. Checked 20 August 2026.PerplexityBot surfaces and links sites in results and is not used for foundation model training.
- Google Search Central. Google's common crawlers. Updated 14 July 2026.Google-Extended does not impact inclusion in Google Search and is not a ranking signal.
- Google Search Console Help. Search generative AI control and Generative AI performance report. June 2026.Opt out control for generative AI features, plus impressions reporting for AI Overviews and AI Mode.
- Jono Alderson. LLMs aren't playing by Google's rules. 7 May 2025.On AI crawlers not rendering pages or executing scripts the way Google does.
Every figure on this page was read on the source page listed above. This page is reviewed monthly and the last checked date is updated when it is.
