AI crawlers and robots.txt: GPTBot, ClaudeBot choices
Hephaestus Technology7 min read

AI crawlers are automated programs that AI companies use to fetch web pages, and robots.txt is the file where you tell each one whether it may visit your site. The key choice for a B2B company is to treat training crawlers such as GPTBot and ClaudeBot separately from search crawlers such as OAI-SearchBot and Claude-SearchBot: blocking the first keeps your content out of model training, while blocking the second removes you from AI search answers.
Key takeaways
- AI crawlers fall into three groups: training crawlers (GPTBot, ClaudeBot), search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User).
- Blocking a training crawler does not remove you from AI search answers; blocking a search crawler does.
- Google-Extended controls Gemini training and grounding, not Google Search or AI Overviews; opting out of AI Overviews uses a separate Search Console control.
- robots.txt is a request, not a lock: the internet standard says its rules "are not a form of access authorization".
- For most B2B companies the sensible default is to allow search crawlers, make a deliberate choice on training crawlers, and review the file yearly.
What are AI crawlers?
An AI crawler is a bot operated by an AI company that requests pages from websites, either to collect training data, to build a search index for an assistant, or to fetch a page live because a user asked about it. Each identifies itself with a user agent name, and that name is what you target in robots.txt.
robots.txt itself is defined by RFC 9309, the Robots Exclusion Protocol, published as an internet standard in September 2022. The standard is clear that its rules "are not a form of access authorization" and that the protocol is not a substitute for real security. Well-behaved crawlers follow it; it cannot force anyone to.
Which AI crawlers matter, and what does each do?
The main providers now publish separate bots for separate purposes. This table summarises their own documentation.
| Bot | Company | Purpose | Effect of blocking |
|---|---|---|---|
| GPTBot | OpenAI | Collects content that may train foundation models | Signals content should not be used for training |
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | Site "will not be shown in ChatGPT search answers" |
| ChatGPT-User | OpenAI | Fetches pages when a user asks | robots.txt "may not apply" |
| ClaudeBot | Anthropic | Collects training data | Future material excluded from training |
| Claude-SearchBot | Anthropic | Indexes content for search quality | May reduce visibility in Claude search |
| Claude-User | Anthropic | Fetches pages for user queries | Prevents retrieval for user questions |
| PerplexityBot | Perplexity | Surfaces sites in Perplexity results; not used for training | Reduces chance of appearing |
| Perplexity-User | Perplexity | Fetches pages for user questions | "Generally ignores robots.txt rules" |
| Google-Extended | Token (not a crawler) controlling Gemini training and grounding | No effect on Google Search inclusion or ranking |
Sources: OpenAI crawler overview, Anthropic help centre, Perplexity bot guide, Google's common crawlers page.
OpenAI adds that GPTBot and OAI-SearchBot settings are independent, and that if both are allowed it may use one crawl for both purposes. Anthropic says its bots honour robots.txt and support the non-standard Crawl-delay directive, and that blocking its IP addresses may not work reliably.
How do Google's AI features fit in?
Google's AI Overviews and AI Mode use the ordinary Googlebot index, so robots.txt choices for AI-specific bots do not take you out of them. Google says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search"; it governs whether Google may use your content to train future Gemini models and for grounding in Gemini Apps and Vertex AI.
To leave Google's generative AI search features, Google has added a Search Console control. Its announcement says sites that opt out "will not receive traffic or impressions" from those features and that the control "will not be used as a ranking signal" for results outside them. Snippet controls such as nosnippet and max-snippet also limit what is shown, per Google's AI features page.
A cross-provider control is coming. On 30 October 2026, Google said it will support a shared AI training preference based on IETF AIPREF work, letting sites set training preferences across participating providers with one robots.txt entry. Google aims to support it in 2027; if a site opts out of training this way without mentioning Google-Extended, Google will treat it as disallowing Google-Extended.
What robots.txt setups make sense for a B2B site?
Three common setups, depending on your priorities. These are illustrative; test any change before deploying.
Setup A: maximum AI visibility (allow everything). Suitable for most service firms that want to be found.
User-agent: *
Allow: /
Setup B: appear in AI search, opt out of training. The most common choice for firms that want citations but not model training.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
OAI-SearchBot, Claude-SearchBot and PerplexityBot fall under the general rule and remain allowed.
Setup C: protect a premium section only. Block AI bots from a paid or gated area while leaving marketing pages open, for example Disallow: /members/ under each AI user agent. Remember that gated content should be protected by login, not robots.txt.
Anthropic notes that rules must be set on each subdomain's own robots.txt file. OpenAI says changes can take about 24 hours to take effect.
What are the common pitfalls?
The most common mistakes come from templates and defaults rather than decisions.
- Blocking search crawlers by accident. A copied "block all AI bots" list often includes OAI-SearchBot or PerplexityBot, which removes you from those assistants' answers.
- CDN defaults. Cloudflare began blocking AI crawlers by default for new domains on 1 July 2025, so a firewall rule may override what your robots.txt says.
- Expecting robots.txt to protect confidential pages. The standard says it is not access authorisation; use authentication.
- Assuming llms.txt controls access. It does not; see our explainer on what llms.txt is.
- Forgetting user-initiated fetchers. ChatGPT-User and Perplexity-User may fetch pages when a user pastes a link, regardless of robots.txt.
How do you check what is happening?
Check your server or CDN logs for the user agents above, and verify suspicious visits against published IP lists; OpenAI publishes ranges for each of its bots. If AI search crawlers never visit your key pages, fix access before any content work. Our guide on how to measure AI search visibility shows how to combine log checks with Search Console and Bing data, and how to get cited by ChatGPT, Claude and Perplexity covers the content side.
Frequently asked questions
If I block GPTBot, will ChatGPT stop mentioning my company?
No. GPTBot only governs whether content may be used to train OpenAI's models. ChatGPT search answers depend on OAI-SearchBot, which is a separate control, and the model may still know about your company from other sources.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal there. It covers Gemini model training and grounding in Gemini Apps and Vertex AI. To opt out of AI Overviews and AI Mode, Google offers a Search Console control instead.
Do AI crawlers really obey robots.txt?
The major providers say their automated crawlers do: OpenAI, Anthropic and Perplexity all document robots.txt support. User-initiated fetchers are different; OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. For anything sensitive, use authentication, not robots.txt.
How quickly do robots.txt changes take effect?
It varies by provider. OpenAI says it can take about 24 hours for its systems to adjust after a robots.txt update. Google notes that recrawling to reflect changes can take from days to months for some controls.
Is blocking AI training crawlers legally required in the EU?
No. Nothing requires a website owner to block AI crawlers; it is a business decision about how your content may be used. If reuse of copyright material is a serious concern for your business, take legal advice rather than relying on robots.txt alone.
If you would like someone to review your crawler settings against your goals, book a free 30-minute call with our team.
Cover photo by Christina Morillo on Pexels.