In July 2025, Cloudflare flipped a switch on the economics of the open web. AI crawlers went from allowed unless blocked to blocked unless permitted for new domains on its network, and a new product — pay-per-crawl — offered a third option between open access and a locked door: name a price. Every content site behind Cloudflare now faces a strategic question: charge AI crawlers, block them, or court them? The answer depends entirely on what your content is for. Here is what pay-per-crawl actually is according to Cloudflare’s own pages, who benefits from charging, who gets hurt, and a decision framework.
What pay-per-crawl is — and its real status in August 2026
Pay-per-crawl is a Cloudflare feature, announced on 1 July 2025, that lets a domain owner set a price for AI crawler access. The mechanics revive a long-dormant corner of the web: HTTP response code 402, “Payment Required”. A crawler that hasn’t agreed to pay receives a 402 response with pricing details; a crawler that presents payment intent via request headers gets the content with a normal 200 response.
For each crawler, a site owner picks one of three actions:
- Allow — free access, no charge;
- Charge — access at the configured price, enforced via HTTP 402;
- Block — no access at any price.
At launch, pricing was deliberately simple: Cloudflare said domain owners “can define a flat, per-request price across their entire site”. Cloudflare also states that it “acts as the Merchant of Record for pay per crawl and also provides the underlying technical infrastructure” — it sits in the middle of the money flow between AI companies and site owners.
Here is the detail that often gets lost in coverage: pay-per-crawl has not shipped to everyone. Cloudflare’s documentation, current as of this writing, states: “Pay per crawl is currently in closed beta.” More than a year after the private beta launch, access still requires signing up and being admitted. If you have read headlines suggesting every site can start invoicing OpenAI tomorrow, that is not yet the reality.
“Every new domain”: scoping the default block precisely
Alongside pay-per-crawl, Cloudflare made a second, arguably bigger change. Its 1 July 2025 press release announced it is “the first Internet infrastructure provider to block AI crawlers accessing content without permission or compensation, by default”, scoped like this: “every new domain starts with the default of control, and eliminates the need for webpage owners to manually configure their settings to opt out.” The same release states that “AI companies will now be required to obtain explicit permission from a website before scraping.”
Read that scope carefully. The default applies to new domains signing up to Cloudflare from July 2025 onward. Existing customers were not force-flipped; their configured settings stand. But as domains churn onto Cloudflare’s network, a growing share of the web defaults to “AI crawlers keep out” — whether or not the site owner ever thought about it. If your marketing site is on Cloudflare and AI visibility matters to you, check what your crawler settings actually say rather than assuming openness. We cover doing that deliberately in our guide to building an AI crawler allowlist in robots.txt for B2B SaaS.
How the product has matured since launch
Cloudflare’s AI Crawl Control changelog shows pay-per-crawl evolving from a blunt instrument into yield management:
- December 2025: a Discovery API so verified crawlers can programmatically find participating domains; Web Bot Auth signatures on payment headers; structured error codes; free pages via Configuration Rules; and always-free paths including /robots.txt and /sitemap.xml, so crawlers can still see your directives without paying.
- June 2026: dynamic pricing — prices set per request via a crawler-price response header from the origin or via Cloudflare Workers — plus the ability to disable pay-per-crawl on URI patterns, so parts of a site stay free while others are metered.
The June 2026 change matters strategically: sites no longer face a single sitewide price. A publisher can charge for its archive while leaving its About page and documentation free. That granularity weakens the strongest argument against charging — that it is all-or-nothing — but it does not change the fundamental trade-off: what is a crawler’s visit worth to you?
The publisher case for charging
Be fair to the charging side, because the case is real. For a news publisher, research house or reference site, content is the product. The old bargain with search crawlers was roughly balanced: crawlers took your content, and search results sent readers back to see your ads or buy subscriptions. AI assistants consume the content and deliver the answer directly, and the reader often never arrives. For a business whose revenue depends on the visit, an AI crawler is a cost with no corresponding benefit.
For that business, pay-per-crawl is a rational response: convert an uncompensated taking into a licensing revenue line, with metering, authentication and settlement provided by a party already sitting in front of the traffic. The 402-based design is also more honest than silent blocking: it tells the crawler exactly what access costs, and lets AI companies make a commercial decision rather than an adversarial scraping one.
The SaaS case for courting crawlers
Now invert every assumption. If you run a B2B SaaS company, your blog posts, documentation and comparison pages are not the product — they are marketing for the product. Nobody pays to read them; they exist so buyers discover, trust and choose your software. An AI crawler that reads that content and cites it in an answer is not extracting value — it is distributing you, in the channel where a growing share of software buying research now happens.
Charge or block those crawlers and the loss is invisible but compounding: your competitors’ pages get ingested, cited and recommended inside AI assistants and AI-augmented search, and yours do not. You will never see a line item for the pipeline that never formed. This is the logic we unpack in our post on zero-click AI search and brand visibility: when the answer layer absorbs the click, being the brand named in the answer is the remaining prize, and crawler access is the price of admission.
A blanket “charge them all” posture also misses a technical nuance: not all AI crawlers do the same job. Training crawlers bulk-ingest content for future models; on-demand fetchers retrieve your page in real time because a user’s question surfaced it — about as high-intent as machine traffic gets. Charging a real-time fetcher is close to charging a prospective customer’s research assistant at the door. We break down the taxonomy in AI crawlers: training bots vs on-demand fetchers.
This is, transparently, the world Zian AI operates in: our autonomous AI sales agents platform grows by being discoverable where buyers ask questions, so we treat crawler access as distribution, not licensing. If you are building a partner practice around AI-driven sales and visibility, Apply For Partnership.
Charge, block or court: a side-by-side comparison
| Charge (pay-per-crawl) | Block | Court (allow & optimise) | |
|---|---|---|---|
| Core goal | Convert crawls into licensing revenue | Deny use of content entirely | Maximise AI citations and mentions |
| Content’s role | Content is the product | Content is proprietary or sensitive | Content is marketing for a product |
| Revenue mechanism | Per-request payment via HTTP 402, Cloudflare as Merchant of Record | None (defensive) | Indirect: pipeline from AI-referred and AI-influenced buyers |
| AI visibility impact | Reduced unless AI firms pay; unpaid crawlers see 402s, not content | Near zero over time as indexes refresh | Maximised; you compete on content quality and structure |
| Availability today | Closed beta (sign-up required) | Default for new Cloudflare domains; available to all | Available to everyone, no gatekeeper |
| Best suited to | Publishers, wire services, reference and data sites | Paywalled archives, regulated or private content | B2B SaaS, professional services, anyone selling something other than the content itself |
| Key risk | AI firms decline to pay and route around you; visibility quietly erodes | Competitors own every AI answer in your category | Giving value to models with no guaranteed citation in return |
A decision framework: five questions before you touch the dial
- Does anyone pay you (directly or via ads) to consume this content? If yes, you sit on the publisher side and charging deserves serious analysis. If no, every crawl you deter is marketing you declined.
- Where do your buyers research? If your category’s buyers ask AI assistants “best X for Y” questions, AI answer visibility is a channel, and crawler access is its supply chain.
- Can you split the difference? Since June 2026, pay-per-crawl supports URI-pattern exemptions and dynamic pricing, so a hybrid — free product pages, priced archive — is feasible for beta participants. Outside the beta, robots.txt lets you allow answer-engine fetchers while disallowing pure training bots.
- What is your default right now? If you joined Cloudflare after 1 July 2025, AI crawlers may be blocked without you ever choosing it. Audit before you strategise.
- What would a paying crawler actually buy? If the honest answer is “pages we wrote to be found”, charging is a tax on your own reach. If it is “a decade of original reporting”, it is a licence fee.
For most B2B SaaS brands, the framework lands in the same place: court, don’t charge. Charging suits businesses that monetise attention on their own pages. If you monetise software, the AI answer layer is a shop window — and shop windows work better unshuttered.
Sources & ownership
| Figure / claim | Owning organisation | Owner URL |
|---|---|---|
| Pay-per-crawl launch (1 July 2025), HTTP 402 mechanism, Allow/Charge/Block actions, “flat, per-request price across their entire site”, Cloudflare as Merchant of Record | Cloudflare | blog.cloudflare.com/introducing-pay-per-crawl |
| Default block scope: “every new domain starts with the default of control”; AI companies “required to obtain explicit permission from a website before scraping” | Cloudflare | cloudflare.com press release, 1 July 2025 |
| “Pay per crawl is currently in closed beta”; per-zone pricing; WAF/Bot Management blocks override charging | Cloudflare | developers.cloudflare.com — What is Pay Per Crawl? |
| December 2025 enhancements (Discovery API, Web Bot Auth signatures, always-free paths incl. /robots.txt and /sitemap.xml); June 2026 dynamic pricing via crawler-price header/Workers and URI-pattern exemptions | Cloudflare | developers.cloudflare.com — AI Crawl Control changelog |
| Google users clicked a traditional result on 8% of visits with an AI summary vs 15% without; clicked a link inside the AI summary on 1% of such visits | Pew Research Center | pewresearch.org, 22 July 2025 |
Frequently asked questions
Is Cloudflare’s pay-per-crawl generally available?
No. As of August 2026, Cloudflare’s documentation states that “Pay per crawl is currently in closed beta.” Site owners and AI crawler operators must sign up and be admitted. The feature set has expanded substantially since the July 2025 private beta launch, but it is not yet open to every Cloudflare customer.
Does Cloudflare’s default AI-crawler block apply to my existing domain?
The default applies to new domains: Cloudflare’s July 2025 announcement says “every new domain starts with the default of control”, removing the need for new sign-ups to opt out manually. Existing customers keep their configured settings and choose their own posture. Either way, audit your configuration — it may not match your strategy.
If AI answers reduce clicks anyway, why court AI crawlers at all?
Because the visibility moves into the answer itself. Pew Research Center found that Google users who encountered an AI summary clicked a traditional search result on just 8% of visits, versus 15% when no summary appeared — and clicked a link inside the summary on only 1% of such visits. When clicks compress like that, being the brand named and cited within AI answers is the visibility that remains, and crawlers can only cite content they can access.
Can I charge some AI crawlers and welcome others?
Yes. Pay-per-crawl is configured per crawler with three actions — Allow, Charge or Block — and since June 2026 beta participants can also exempt URI patterns and set dynamic per-request prices. Outside the beta, robots.txt directives let any site allow on-demand answer fetchers while disallowing training-only bots, which suits brands that want citations without contributing to model training.
Does pay-per-crawl replace robots.txt?
No. They operate at different layers: robots.txt is a voluntary directive that well-behaved crawlers read, while pay-per-crawl enforces payment at the network edge with HTTP 402 responses. Cloudflare keeps paths such as /robots.txt and /sitemap.xml always free within pay-per-crawl so crawlers can still read your directives, and existing WAF or Bot Management blocks override any charge setting.
Zian AI builds autonomous AI sales agents that thrive on the same answer-engine visibility this post describes — SmartReach AI™ orchestrating outreach across phone, SMS, email and WhatsApp, and PrecisionPitch AI™ continuously split-testing what works. If that intersection is where you play, Apply For Partnership.