Field note
Which AI crawlers should you allow
If you want to be cited by an answer engine, its crawler has to be able to reach you. Most people have never checked, and a surprising number are blocking the exact systems they want to appear in.
More than five years in personal branding. Runs atharvsankpal.com and works with founders and businesses in the US on their digital presence. Previously produced AI video for brands.
Published 2026-08-14
Researched against primary sources, drafted with AI assistance, then reviewed and approved by the named author. How we write · About the author
AI crawlers fall into two rough groups, and conflating them is the most common mistake. Retrieval crawlers fetch pages at the moment someone asks a question, so blocking them directly prevents citation. Training crawlers collect material for model training, and blocking those is a legitimate licensing decision that does not stop you being cited in live answers. If your goal is to be named when someone asks an answer engine a question, the retrieval agents need access. Two traps are worth knowing: managed bot-blocking features at the CDN or host layer can add rules on top of your own file without touching your code, and a file containing both an Allow and a Disallow for the same agent is resolved differently by different parsers.
- Retrieval and training are different decisions. Blocking one is not blocking the other.
- If you want citation, the retrieval agents need access. That is not optional.
- Managed CDN bot rules can inject blocks your own file never contained.
- Contradictory rules are not safe. Parsers disagree, so the outcome is undefined.
The two kinds of crawler
The distinction that matters is what the fetch is for.
Retrieval. Someone asks a question, the system searches, fetches pages and composes an answer citing them. If your site is blocked to that agent, you cannot appear in that answer. This is the group to allow if being cited is the goal.
Training. Material is collected to train or improve a model. Blocking here is a rights and licensing decision. Plenty of publishers block training and allow retrieval, which is a coherent position: be quotable in live answers, without contributing to a training corpus.
Treating these as one decision is how people end up blocking their own visibility while intending to protect their content.
Who is who
| Agent | Broadly does | Block it if |
|---|---|---|
| GPTBot | Crawls for OpenAI, associated with training | You object to training use |
| OAI-SearchBot | Indexes for OpenAI's search features | You do not want to appear in those answers |
| ChatGPT-User | Fetches a page because a user's request needs it | Rarely worth blocking |
| ClaudeBot | Crawls for Anthropic | You object to training use |
| Claude-User | Fetches on behalf of a user request | Rarely worth blocking |
| Claude-SearchBot | Crawls to improve the quality of Claude’s search results | You do not want to be cited in Claude answers |
| PerplexityBot | Crawls for Perplexity's answers | You do not want to be cited there |
| Google-Extended | Controls use in Google's AI products, not normal Search ranking | You object to that use |
| Applebot-Extended | Controls AI training use for Apple | You object to training use |
Worth being precise about one of these: Google-Extended governs whether your content is used in Google's generative AI products. It is not a Search ranking control, and blocking it does not remove you from Google Search.
The two traps
Rules you did not write. Many CDNs and hosting platforms now ship managed bot-control features that add AI crawler rules automatically. These are applied at the edge, which means your repository's file can be perfectly correct while the file actually served contains additional blocks. Checking the deployed file rather than the source file is the only way to know.
Contradictory rules. If the same agent appears in one group with a disallow and another with an allow, both covering the whole site, the standard leaves the tie to the implementation. Some parsers combine the groups and resolve in favor of the less restrictive rule, and some take the first matching group and stop. The honest description of that file is that its effect is undefined and varies by crawler, which is not a state you want for the systems you depend on.
How to check
- Fetch the live file at your domain followed by /robots.txt and read what is actually served, not what is in your repository.
- Look for any agent appearing more than once with conflicting directives, and remove the duplication rather than reasoning about precedence.
- Check whether your host or CDN has a bot-management or AI crawler feature enabled, and whether it injects rules.
- Confirm your pages render without JavaScript, since some retrieval systems will otherwise see very little.
- Re-check after platform updates, because managed rules can reappear without any change on your side.
Questions people actually ask
Does blocking AI crawlers hurt my visibility?
It depends which ones. Blocking retrieval crawlers directly prevents you being cited in live AI answers. Blocking training crawlers is a licensing decision that does not stop you appearing in retrieval-based answers.
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended governs use of your content in Google's generative AI products. Normal Search crawling and ranking are controlled separately by Googlebot.
What if my robots.txt has both an allow and a disallow for the same bot?
Then the outcome is genuinely undefined. Parsers differ: some merge the groups and favor the less restrictive rule, others take the first match and stop. The fix is to remove the duplication rather than to work out which interpretation wins.
Why would my robots.txt contain rules I did not write?
Because many CDNs and hosting platforms add AI crawler rules at the edge through managed bot-control features. The deployed file can differ from the one in your repository, which is why checking the live file matters.
References
- OpenAI. Overview of OpenAI crawlers — GPTBot, OAI-SearchBot and ChatGPT-User.
- Anthropic. Crawler documentation — ClaudeBot, Claude-User and Claude-SearchBot.
- Google Search Central. Crawlers and user agents, including the Google-Extended control.
- RFC 9309. Robots Exclusion Protocol.
How this was made: researched against primary sources, drafted with AI assistance, then reviewed and approved by the named author before publication. Our editorial standards.
Access is only the first gate. What you publish decides whether you are worth quoting once they arrive.
How answer engines pick sources →Related reading
Want your access checked properly?
We will fetch what is actually being served, find rules you did not write, and tell you what is blocking you from the systems you want to appear in.
Book a 30 minute call