Somewhere in the last two years, a security checklist probably told you to block AI crawlers in robots.txt. For a B2B SaaS company in 2026, that advice can quietly cost you the channel where buyers now do their research — because the same AI company typically operates one crawler that feeds model training and a different one that decides whether you appear in the answers a buyer reads. This post is the decision framework: who commits to honouring robots.txt (verified at each owner’s own documentation), which agents to allow, which to block, and what to do about crawlers that ignore the file entirely.
At a glance: Blocking a training crawler like GPTBot keeps your content out of future model training, but blocking OAI-SearchBot or ChatGPT-User removes you from ChatGPT’s search answers — the surface your buyers actually read. For most B2B SaaS sites, the sane 2026 default is: allow search and user-triggered agents, decide training bots case by case, and remember robots.txt is voluntary — real enforcement happens at the CDN or firewall, not in the text file.
One company, several bots — and only some of them matter to revenue
Each major AI company now runs multiple agents with different jobs. Broadly: training crawlers collect content for model training, search crawlers build the index AI answers cite, and user-triggered fetchers retrieve a page live because a human just asked about it. (This post assumes you know who is knocking — identification is a separate exercise — and focuses on whether to let them in.)
The revenue-relevant distinction: training crawlers give you nothing directly. Search crawlers and user fetchers are how an assistant cites your comparison post or documentation to a buyer mid-evaluation. Block the wrong class and you have opted out of being the answer.
Who actually commits to honouring robots.txt
robots.txt is the Robots Exclusion Protocol — a convention, not an access control. Whether a directive does anything depends entirely on whether the operator chooses to honour it. So the first question is not “what should I write?” but “who has publicly committed to obeying what I write?” Here is what each owner says in its own documentation, as of August 2026:
| Owner | Agent (robots.txt token) | Documented purpose | Owner’s stated robots.txt position |
|---|---|---|---|
| OpenAI | GPTBot |
Crawls “content that may be used in training” | Respects robots.txt |
| OpenAI | OAI-SearchBot |
Used “to surface websites in search results in ChatGPT’s search features” | Respects robots.txt |
| OpenAI | ChatGPT-User |
User-initiated page visits; “not used for crawling the web in an automatic fashion” | OpenAI says robots.txt rules “may not apply” to user-triggered actions |
| Anthropic | ClaudeBot |
Collects web content that may contribute to model training | “Anthropic’s Bots respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt” |
| Anthropic | Claude-SearchBot, Claude-User |
Search-quality crawling; user-requested fetches | Same commitment covers all three Anthropic agents |
| Perplexity | PerplexityBot |
Surfaces and links websites in Perplexity search results | Respects robots.txt |
| Perplexity | Perplexity-User |
Visits pages when a user asks a question | Perplexity’s own docs: “this fetcher generally ignores robots.txt rules” |
Google-Extended |
Controls use of your content for Gemini training and grounding — not a separate crawler, and not Google Search | Honoured as a robots.txt token; explicitly “does not impact a site’s inclusion in Google Search” | |
| Meta | meta-externalagent |
“training foundation AI models or improving products by indexing content directly” | Respects robots.txt |
| Meta | meta-externalfetcher |
User-initiated fetches for agentic AI features | Meta’s own docs: “may bypass robots.txt rules” |
| Amazon | Amazonbot |
Improves products and services; “may be used to train Amazon AI models” | Honours allow/disallow; does not support crawl-delay |
| Amazon | Amzn-SearchBot, Amzn-User |
Search experiences (e.g. Alexa); user-triggered fetches | SearchBot follows robots.txt; Amzn-User “may not follow all robots.txt directives” |
| ByteDance | Bytespider |
No official ByteDance crawler documentation page we could locate | No published commitment to honour robots.txt |
Two patterns jump out. Every owner’s automated crawlers come with a compliance commitment — those are the ones you can genuinely steer with robots.txt. And every owner’s user-triggered fetcher comes with a disclaimer: OpenAI, Perplexity, Meta and Amazon all state in their own documentation that when a human asks for a page, the fetch may ignore robots.txt. Not a scandal — just the owners saying honestly that robots.txt governs crawling, not a user’s individual request. But it means a robots.txt-only strategy was never airtight.
The trade-off the checklist missed
Here is the asymmetry that makes blanket blocking self-harm for most B2B SaaS sites. Blocking GPTBot costs you almost nothing visible: your content stays out of future OpenAI training runs, and your ChatGPT search presence is unaffected, because OpenAI documents search surfacing as OAI-SearchBot‘s job, not GPTBot’s. But a blanket rule that also catches OAI-SearchBot and ChatGPT-User tells OpenAI’s search index to exclude you — precisely when a growing share of software evaluation happens inside assistants rather than on results pages.
Google splits the same decision even more cleanly. Google-Extended is a robots.txt token (not a crawler with its own user-agent string) that only controls whether your content trains and grounds Gemini. Google Search itself — including AI Overviews and AI Mode — runs on ordinary Googlebot crawling, and Google states that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search”. To limit what AI Overviews shows without leaving Search, the levers are nosnippet, data-nosnippet, max-snippet and noindex — not robots.txt. You can decline Gemini training at zero search cost; you cannot decline AI Overviews at zero search cost.
So the honest question is narrow: is keeping public marketing pages out of training corpora worth anything to you? For proprietary datasets or gated research — often yes. For the posts and docs you publish precisely so buyers and their tools will find them, training-bot blocking is a defensible preference; search-bot blocking is a visibility own-goal.
A decision matrix by site type and goal
| Site / goal | Training bots (GPTBot, ClaudeBot, meta-externalagent) | Search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Amzn-SearchBot) | User fetchers (ChatGPT-User, Claude-User, Perplexity-User) | Google-Extended |
|---|---|---|---|---|
| B2B SaaS marketing site — goal: be cited in buyer research | Your call — blocking costs no visibility | Allow | Allow (blocking is mostly symbolic anyway) | Your call — no Search impact either way |
| Docs / knowledge base — goal: assistants answer support questions correctly | Allow (trained-in knowledge helps accuracy) | Allow | Allow | Allow |
| Gated / premium research — goal: protect the asset | Block | Block (you don’t want teaser pages answering the whole question) | Block in robots.txt, but enforce at the edge | Block |
| Publisher / content-is-the-product — goal: compensation before access | Block | Case by case (citations vs cannibalisation) | Edge enforcement + licensing conversation | Block |
| App / customer portal — goal: no bots at all | Block everything and enforce with authentication — robots.txt is not a security layer | Same | Same | Same |
A copy-paste allowlist for the common SaaS case
This block implements the most common B2B SaaS position: visible in AI answers, opted out of training. Adjust to taste — the comments explain what each stanza costs or protects. One gotcha first: robots.txt matching is per-group. A bot that finds its own User-agent group ignores your User-agent: * rules entirely, so listing an agent means you must state all the rules you want it to follow.
# --- AI search crawlers: ALLOW (this is where buyers read about you) ---
User-agent: OAI-SearchBot # ChatGPT search surfacing (OpenAI)
Allow: /
User-agent: PerplexityBot # Perplexity search results
Allow: /
User-agent: Claude-SearchBot # Claude search-quality crawling (Anthropic)
Allow: /
User-agent: Amzn-SearchBot # Amazon search experiences, e.g. Alexa
Allow: /
# --- User-triggered fetchers: ALLOW (owners say robots.txt may not bind them anyway) ---
User-agent: ChatGPT-User # live fetch when a ChatGPT user asks about you
Allow: /
User-agent: Claude-User # live fetch for Claude users
Allow: /
# --- Training crawlers: BLOCK (no visibility cost; content stays out of training) ---
User-agent: GPTBot # OpenAI training corpus crawler
Disallow: /
User-agent: ClaudeBot # Anthropic training-related crawler
Disallow: /
User-agent: meta-externalagent # Meta AI training / index crawler
Disallow: /
User-agent: Amazonbot # may train Amazon AI models; no crawl-delay support
Disallow: /
User-agent: Google-Extended # token only: opts out of Gemini training/grounding,
Disallow: / # does NOT affect Google Search or rankings
User-agent: Bytespider # ByteDance; no published compliance commitment —
Disallow: / # keep the line for the record, enforce at the edge
Note what is absent: no Googlebot rules (blocking it removes you from Google Search itself), and no illusion that the Bytespider stanza does anything on its own.
robots.txt is the policy; the CDN is the police
Because the protocol is voluntary, compliance is an empirical question — and the best public evidence comes from networks sitting in front of millions of sites. In August 2025, Cloudflare published research finding that Perplexity was using “stealth, undeclared crawlers” to reach content after its declared bots were blocked: a generic Chrome-like user agent, rotating IPs and shifting network origins in response to restrictive robots.txt policies. Cloudflare de-listed Perplexity as a verified bot in response. Whatever you make of the dispute, the structural lesson stands: a text file cannot stop a crawler that has decided not to read it.
That is why the enforcement layer matters. Cloudflare’s AI Crawl Control (available on all plans) shows which AI services access your content, sets allow or block rules per crawler, and tracks which crawlers actually follow your robots.txt directives so you can create enforcement rules for those that do not; a pay-per-crawl option is in private beta. Cloudflare also announced in July 2025 that it was “changing the default to block AI crawlers unless they pay creators for their content” — per Cloudflare’s press release of the same day, “every new domain starts with the default of control”: new domains signing up are asked whether to allow AI crawlers, and blocking is the starting position. If your marketing site went behind Cloudflare recently, check that default before wondering why your AI-search citations dried up: your enforcement layer may be blocking the very search bots your robots.txt allows.
The working model has three layers: robots.txt states your policy; your CDN or WAF enforces it against non-compliant crawlers; and authentication protects anything that must not leak — because neither of the first two is a security control.
FAQ
Does blocking GPTBot remove my site from ChatGPT’s answers?
No. OpenAI documents these as separate agents: GPTBot crawls content that may be used in training, while OpenAI’s crawler documentation states that OAI-SearchBot is used “to surface websites in search results in ChatGPT’s search features” and that ChatGPT-User handles user-initiated visits. Block GPTBot alone and your ChatGPT search visibility is unaffected.
Will blocking Google-Extended hurt my Google rankings?
No. Google Search Central’s crawler documentation states that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search”. It only controls whether your content is used for Gemini training and grounding.
Can robots.txt actually stop a crawler that ignores it?
No — it is a voluntary protocol. Cloudflare’s August 2025 research documented crawling that continued via undeclared user agents and rotating IPs after declared bots were blocked. If a directive must hold, enforce it at the CDN, WAF or server level.
Should we block user-triggered fetchers like ChatGPT-User or Perplexity-User?
Usually not, for two reasons. Strategically, these fetches happen because a buyer just asked about you — blocking them degrades the answer that buyer receives. Practically, the owners themselves say robots.txt may not bind these agents: Perplexity’s docs state Perplexity-User “generally ignores robots.txt rules”, and OpenAI and Meta publish similar caveats for their user-triggered agents. If you genuinely need to stop them, that is an edge-enforcement job.
We blocked everything a year ago. Is the damage permanent?
For search crawlers, no — unblock the search agents and they can recrawl and re-index your pages, though recovery is not instant and AI search indexes refresh on their own schedules. For training crawlers, blocking is forward-looking either way: robots.txt affects future crawls, not content already collected.
How often should we revisit these decisions?
Quarterly, or whenever an owner announces a new agent — a robots.txt written for last year’s bot names silently fails to cover this year’s.
If your buyers research in chatbots — and for B2B SaaS in 2026, a meaningful share do — the crawler allowlist is now part of your funnel plumbing. Zian’s agents are built to work the other side of that funnel: engaging, qualifying and booking the buyers your visibility work attracts, across phone, SMS, email and WhatsApp.
Sources and ownership
Every external claim above was verified at the owning organisation’s own page:
| Claim / figure | Owner | Verified at |
|---|---|---|
| GPTBot, OAI-SearchBot, ChatGPT-User purposes; robots.txt positions | OpenAI | https://developers.openai.com/api/docs/bots |
| ClaudeBot, Claude-SearchBot, Claude-User; “respect ‘do not crawl’ signals” quote | Anthropic | https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler |
| PerplexityBot; Perplexity-User “generally ignores robots.txt rules” | Perplexity | https://docs.perplexity.ai/guides/bots |
| Google-Extended scope; “does not impact a site’s inclusion in Google Search” quote | https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers | |
| AI Overviews/AI Mode use Googlebot; nosnippet family as the preview controls | https://developers.google.com/search/docs/appearance/ai-features | |
| meta-externalagent; meta-externalfetcher “may bypass robots.txt rules” | Meta | https://developers.facebook.com/docs/sharing/webmasters/web-crawlers |
| Amazonbot, Amzn-SearchBot, Amzn-User roles; no crawl-delay support | Amazon | https://developer.amazon.com/en/amazonbot |
| AI Crawl Control features and plan availability | Cloudflare | https://developers.cloudflare.com/ai-crawl-control/ |
| Perplexity stealth-crawling findings; de-listing (4 August 2025) | Cloudflare | https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ |
| “changing the default to block AI crawlers unless they pay creators for their content” (1 July 2025) | Cloudflare | https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/ |
| Block-by-default scope for new domains: “every new domain starts with the default of control” | Cloudflare | https://www.cloudflare.com/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/ |
ByteDance’s Bytespider: we could locate no official ByteDance crawler documentation page, so no compliance commitment is attributed to it above — that absence is itself the finding.