Open a server log on any B2B site in 2026 and you will find a crowd of user agents with AI-sounding names: GPTBot, ClaudeBot, PerplexityBot, meta-externalagent, Amazonbot, Bytespider. Most teams treat them as one blob called “AI bots” and either block the lot or ignore the lot. Both reactions miss the point, because these crawlers do two completely different jobs — and only one involves a potential buyer.
At a glance: AI crawlers split into two families. Training crawlers (GPTBot, ClaudeBot, meta-externalagent, Amazonbot, CCBot, Bytespider) collect content that may feed a future model — nobody is reading your page today. On-demand crawlers — search indexers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and user-triggered fetchers (ChatGPT-User, Perplexity-User, Claude-User) — fetch or index your page so an AI assistant can cite it to a real person, sometimes right now, mid-conversation. Identify each bot from its owner’s documentation, verify identity against owner-published IP ranges (user-agent strings are trivially spoofed), and only then decide policy.
Why the split matters to a revenue team
A training crawler is a librarian photocopying your site for an archive that might inform a model shipped next year. Whether that is acceptable is a licensing and IP question, and reasonable businesses land on both sides.
An on-demand fetcher is different in kind. When ChatGPT-User or Perplexity-User appears in your logs, a human — possibly a buyer mid-evaluation — asked an AI assistant a question, and the assistant is fetching your page to answer them in that moment. In server logs across the industry, these fetches cluster around pricing pages, comparison posts and documentation: exactly the pages prospects ask assistants about. Blocking a training bot costs inclusion in a future corpus. Blocking an on-demand fetcher can cost a live citation in front of a live buyer.
The catch: both families ship from the same companies with confusingly similar names, and the only authoritative description of each bot is the owner’s own documentation — third-party “bot list” blogs routinely conflate them. Everything below is verified against the operator’s published page, and where an operator publishes nothing, we say so, because that is part of the story.
The training crawlers: feeding a future model
These bots crawl broadly and continuously. Their owners say, with varying candour, that the content may be used to train models.
- GPTBot (OpenAI) — may “crawl content that may be used in training our generative AI foundation models”. Respects robots.txt; IP ranges published at openai.com/gptbot.json.
- ClaudeBot (Anthropic) — “helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.” Anthropic says its bots “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt” and, unusually, also supports the non-standard
Crawl-delayextension. - meta-externalagent (Meta) — “crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.” Respects robots.txt per Meta’s docs.
- Amazonbot (Amazon) — “used to improve our products and services… and may be used to train Amazon AI models.” Honours robots.txt allow/disallow and page-level meta tags, but does not support
crawl-delay. Amazon also documents two separate non-training agents, Amzn-SearchBot and Amzn-User (below). - CCBot (Common Crawl) — a non-profit building “an open repository of web crawl data that is universally accessible and analyzable by anyone.” Many foundation models train on Common Crawl-derived datasets, so a CCBot decision covers every downstream model builder using the corpus. Respects robots.txt; IPs at index.commoncrawl.org/ccbot.json.
- Bytespider (ByteDance) — the outlier. ByteDance publishes no English-language bot documentation: no purpose statement, no robots.txt compliance commitment, no IP ranges. Its owner-side presence is the Chinese-language Toutiao Search webmaster platform (zhanzhang.toutiao.com); the user-agent seen in logs identifies itself only with a contact email. When an operator publishes nothing, you cannot verify its traffic or hold it to any policy — which is itself the most important thing to know about it.
The on-demand crawlers: a buyer may be on the other end
Owner documentation splits this family into two sub-types, and the distinction matters.
Search indexers — they decide whether you’re findable
- OAI-SearchBot (OpenAI) — used to “surface websites in search results in ChatGPT’s search features”; OpenAI tells site owners to “use OAI-SearchBot in robots.txt for managing Search opt outs”. Respects robots.txt.
- PerplexityBot (Perplexity) — “designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.” Perplexity recommends allowing it in robots.txt, but its docs stop short of an explicit compliance statement.
- Claude-SearchBot (Anthropic) — “navigates the web to improve search result quality for users.” Respects robots.txt.
- Meta-WebIndexer (Meta) — indexes for Meta AI search; allowing it “helps us cite and link to your content in Meta AI’s responses.”
- Amzn-SearchBot (Amazon) — makes content “eligible to appear in search experiences such as Alexa” and “does not crawl content for generative AI model training.”
User-triggered fetchers — someone is reading you right now
- ChatGPT-User (OpenAI) — used “for certain user actions in ChatGPT and Custom GPTs”. OpenAI is candid: “Because these actions are initiated by a user, robots.txt rules may not apply.”
- Perplexity-User (Perplexity) — “supports user actions within Perplexity”; because a person requested the page, this fetcher “generally ignores robots.txt rules”.
- Claude-User (Anthropic) — “When individuals ask questions to Claude, it may access websites using a Claude-User agent.” Anthropic says its bots honour robots.txt directives.
- meta-externalfetcher (Meta) — “fetches individual links at a user’s request and supports product functions such as evaluating and improving agentic AI capabilities… Accordingly, this crawler may bypass robots.txt rules.”
- Amzn-User (Amazon) — fetches live information for Alexa answers; “it may not follow all robots.txt directives” and “does not crawl content for generative AI model training.”
Notice the pattern in the owners’ own words: most operators state plainly that user-triggered fetchers may not obey robots.txt, on the logic that a human asked for the page — the fetch is closer to a browser visit than a crawl.
This taxonomy is the first half of deciding which visitors to welcome. Zian’s autonomous sales agents are built to work the other side of the same shift — engaging the human buyers those AI answers send your way. Apply For Partnership if you want that end of the funnel handled.
The Google exception: one crawler, two knobs
Google refuses to fit the two-family model, and getting this wrong is the most common mistake in AI-crawler policy.
Google-Extended is not a crawler. Per Google’s crawler documentation it is “a standalone product token” — a robots.txt name with no user agent of its own. It controls whether content “may be used for training future generations of Gemini models… and for grounding”. Disallowing it never shows in your logs, because nothing named Google-Extended ever visits.
Googlebot feeds AI Overviews and AI Mode. Google’s AI-features documentation says AI is “integral to how Search functions, which is why robots.txt directives for Googlebot is the control” — and that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”. Translation: you can block Gemini training via Google-Extended while remaining fully visible in AI Overviews; the only way out of AI Overviews is the same machinery as classic search (nosnippet, noindex), with the ordinary visibility cost that implies.
Reference table: every bot, verified at the owner
| Bot (UA token) | Operator | Purpose (per owner docs) | Respects robots.txt (per owner docs) | Why a B2B site cares |
|---|---|---|---|---|
| GPTBot | OpenAI | Training crawl for “generative AI foundation models” | Yes | Future-model inclusion; no live reader behind it |
| ClaudeBot | Anthropic | Training crawl for Claude models | Yes, incl. Crawl-delay | Same trade as GPTBot, for Claude |
| meta-externalagent | Meta | “Training foundation AI models or improving products by indexing” | Yes | Meta AI training corpus; often high request volume |
| Amazonbot | Amazon | Product improvement; “may be used to train Amazon AI models” | Yes (no crawl-delay) | Mixed-purpose; heavy crawler on many sites |
| CCBot | Common Crawl | Open web archive used by many model builders | Yes | One decision covers many downstream trainers |
| Bytespider | ByteDance | No owner documentation published | No owner statement | Unverifiable; UA-only identification is unsafe |
| OAI-SearchBot | OpenAI | Indexing for ChatGPT search | Yes | Gatekeeper for ChatGPT search citations |
| PerplexityBot | Perplexity | Perplexity search index; “not used… for AI foundation models” | Not stated (docs recommend allowing it) | Gatekeeper for Perplexity citations |
| Claude-SearchBot | Anthropic | Improve Claude search result quality | Yes | Findability in Claude’s search |
| Meta-WebIndexer | Meta | Meta AI search index; enables cite/link | Yes (block via disallow) | Citations in Meta AI answers |
| Amzn-SearchBot | Amazon | Alexa search experiences; no gen-AI training | Yes (follows other-search-bot rules if unnamed) | Alexa answer eligibility |
| ChatGPT-User | OpenAI | User actions in ChatGPT / Custom GPTs | “May not apply” (user-initiated) | A live reader via ChatGPT — often a researching buyer |
| Perplexity-User | Perplexity | User actions in Perplexity | “Generally ignores” (user-initiated) | A live reader via Perplexity |
| Claude-User | Anthropic | Fetches pages when a Claude user asks | Yes | A live reader via Claude |
| meta-externalfetcher | Meta | User-requested link fetch; agentic AI tasks | “May bypass” | AI agents completing tasks on your site |
| Amzn-User | Amazon | Live fetches for Alexa user answers | “May not follow all” | A live reader via Alexa |
| Google-Extended | Robots token only — Gemini training + grounding opt-out | n/a (no crawler exists) | Never appears in logs; policy lever only | |
| Googlebot | Search crawl; also feeds AI Overviews / AI Mode | Yes | Blocking it exits search and AI Overviews |
Spoofing: why the user-agent string proves nothing
A user-agent header is a free-text field. Vulnerability scanners and scrapers routinely announce themselves as GPTBot or ChatGPT-User precisely because many sites now allowlist those names — and, in server logs across the industry, “AI crawler” hits probing paths like /.env or /wp-login.php are a reliable tell the UA is fake. Common Crawl says outright it knows of impostors using CCBot’s name.
The fix: most major operators publish machine-readable IP ranges, so identity can be verified rather than trusted.
- OpenAI: per-bot JSON lists — gptbot.json, searchbot.json, chatgpt-user.json.
- Anthropic: combined list at claude.com/crawling/bots.json.
- Perplexity: perplexitybot.json and perplexity-user.json.
- Google: reverse DNS (hostnames under googlebot.com, google.com or googleusercontent.com) plus CIDR files including
common-crawlers.jsonanduser-triggered-fetchers.json. - Amazon: IP ranges on the Amazonbot developer page.
- Common Crawl: reverse DNS under
crawl.commoncrawl.orgplus a JSON IP list. - ByteDance: publishes no verification method at all.
Practical rule: never report “GPTBot crawled us 400 times” from UA strings alone. Filter to successful responses and check a sample of source IPs against the owner’s list first; inflated bot counts lead teams to block things that were never the real bot.
Identification first, policy second
Everything above is about knowing who is at the door. What to do about each visitor — which trainers to allow, whether blocking a fetcher that ignores robots.txt is even meaningful, how to structure an allowlist — is a separate decision with real trade-offs, covered in a companion post on robots.txt strategy for AI crawlers. The short version: you cannot make a sane allow/block call until you can tell a training bot from the fetcher showing your pricing page to a buyer.
Zian exists because a growing share of B2B buying journeys now start inside an AI answer rather than a search results page — and the businesses that win them are ready to respond the moment that buyer raises a hand. Apply For Partnership to put autonomous sales agents on the receiving end of your AI-sourced demand.
FAQ
What is the difference between a training crawler and an on-demand fetcher?
A training crawler (GPTBot, ClaudeBot, meta-externalagent) collects content that may train a future AI model; no human sees your page at crawl time. An on-demand fetcher (ChatGPT-User, Perplexity-User, Claude-User) retrieves a page because a real user just asked an AI assistant something your page can answer — a live reading, often by a researching buyer.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google’s documentation states that Google-Extended controls training and grounding for Gemini models and “does not impact a site’s inclusion in Google Search”. AI Overviews and AI Mode are fed by Googlebot’s ordinary search crawl, so the only opt-outs are standard search controls like nosnippet and noindex — which also affect classic search visibility.
How do I verify a hit claiming to be GPTBot is really OpenAI?
Check the source IP against OpenAI’s published range file (openai.com/gptbot.json). Every major operator except ByteDance publishes IP ranges or a reverse-DNS scheme; a “GPTBot” request from an unlisted IP is a scanner in a costume, not OpenAI.
Which AI bots ignore robots.txt — and is that documented?
By their owners’ own documentation: OpenAI says robots.txt “may not apply” to ChatGPT-User, Perplexity says Perplexity-User “generally ignores robots.txt rules”, Meta says meta-externalfetcher “may bypass robots.txt rules”, and Amazon says Amzn-User “may not follow all robots.txt directives” — all on the rationale that a human user requested the specific page. ByteDance’s Bytespider publishes no compliance statement at all.
How much AI crawler traffic is normal in 2026?
Substantial and growing. Cloudflare’s July 2025 crawler study found AI and search crawler traffic grew 18% from May 2024 to May 2025, with GPTBot’s share of AI-crawler requests rising from 5% to 30% while Bytespider’s fell from 42% to 7% — and of the top-10,000 domains where Cloudflare could find a robots.txt file, only about 14% (546 of 3,816) had directives addressing AI bots at all.
Source-ownership table
| Claim / figure used | Owning organisation | Verified at |
|---|---|---|
| GPTBot, OAI-SearchBot, ChatGPT-User purposes; ChatGPT-User “robots.txt rules may not apply” | OpenAI | https://developers.openai.com/api/docs/bots |
| OpenAI per-bot IP range files | OpenAI | https://openai.com/gptbot.json (also searchbot.json, chatgpt-user.json) |
| ClaudeBot / Claude-User / Claude-SearchBot purposes; robots.txt and Crawl-delay support | Anthropic | https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler |
| Anthropic IP list | Anthropic | https://claude.com/crawling/bots.json |
| PerplexityBot / Perplexity-User purposes; “generally ignores robots.txt rules”; IP files | Perplexity | https://docs.perplexity.ai/guides/bots |
| Google-Extended as product token controlling Gemini training/grounding; no ranking impact | https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers | |
| Googlebot as the control for AI Overviews / AI Mode; snippet controls | https://developers.google.com/search/docs/appearance/ai-features | |
| Google crawler verification (reverse DNS suffixes, CIDR JSON files) | https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot | |
| Amazonbot / Amzn-SearchBot / Amzn-User purposes, robots.txt behaviour, IP ranges | Amazon | https://developer.amazon.com/amazonbot |
| meta-externalagent / meta-externalfetcher / Meta-WebIndexer purposes and robots.txt behaviour | Meta | https://developers.facebook.com/docs/sharing/webmasters/web-crawlers |
| CCBot purpose, robots.txt blocking, impostor warning, IP list | Common Crawl | https://commoncrawl.org/ccbot |
| AI and search crawler traffic +18% (May 2024–May 2025); GPTBot share of AI-crawler requests 5%→30%; Bytespider 42%→7%; ~14% (546 of 3,816) of top-10k domains with a findable robots.txt had AI-bot rules | Cloudflare | https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/ |
| Bytespider: no English owner documentation, no compliance statement, no published IP ranges (statement of absence) | ByteDance (absence verified) | https://zhanzhang.toutiao.com/ (Chinese-language webmaster platform; no bot policy page) |