Web Bot Auth: Cryptographically Signed AI Agents and the End of User-Agent Guessing

At a glance: Web Bot Auth is a proposed method for AI agents to cryptographically sign their HTTP requests, so a website can verify who is calling instead of trusting a user-agent string. As at 26 August 2026 an IETF working group exists but has adopted no document yet, while Cloudflare already ships an implementation. For most site operators it is useful today only through a CDN that verifies signatures on your behalf.

The problem: a user-agent string is just text

Anyone can send ChatGPT-User/1.0 in a header. There is no check, no key, nothing to forge. That mattered little when the worst outcome was a skewed analytics report. It matters now, because operators make real access decisions on the strength of that string: allow this crawler, block that one, serve the paywall or don’t.

How often is that badge worn dishonestly? Nobody publishes a clean global figure, and any site owner quoting their own logs is quoting an unauditable sample — so treat single numbers with suspicion. What is documented is that it happens at scale. On 4 August 2025 Cloudflare published an investigation finding that Perplexity “uses not only their declared user-agent, but also a generic browser intended to impersonate Google Chrome on macOS when their declared crawler was blocked”, using “multiple IPs not listed in Perplexity’s official IP range” and rotating through them, plus requests from different ASNs, “across tens of thousands of domains and millions of requests per day” (Cloudflare, Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives).

Two cautions before you go auditing your own access log. Published IP lists lag reality: vendors add and retire addresses, and the JSON files — OpenAI’s gptbot.json, searchbot.json and chatgpt-user.json, for instance — are snapshots you have to refetch. And “unverified” is not “proven fake”: traffic legitimately traverses proxies and intermediaries, and a miss against a stale list proves only that the list missed it. A failed check is a reason to look harder, not a verdict.

What Web Bot Auth actually is

Web Bot Auth is a scheme in which an automated client signs its outbound HTTP requests with a private key, and the receiving server verifies that signature against a public key the client publishes. It is built on RFC 9421, HTTP Message Signatures, published February 2024 as a Standards Track RFC, which describes “a mechanism for creating, encoding, and verifying digital signatures or message authentication codes over components of an HTTP message”. RFC 9421 is the settled foundation. Web Bot Auth is the bot-specific profile layered on top, and that part is not settled.

The current protocol document is draft-meunier-webbotauth-httpsig-protocol-02, revised 18 August 2026 and due to expire 19 February 2027, authored by Thibault Meunier of Cloudflare and Sandor Major of Google. Mechanically it defines:

• A Signature-Agent request header — a dictionary structured header whose values are HTTPS URIs pointing at where the agent’s key material lives.
• A key directory served at the well-known path /.well-known/http-message-signatures-directory, with the media type application/http-message-signatures-directory+json.
• Required signature parameters: created and expires for time bounds, keyid carrying a base64url JWK SHA-256 thumbprint, and tag set to "web-bot-auth".

The draft is admirably blunt about its own limits. It states that the protocol “does not authenticate human users, does not provide anonymous authentication, and does not define authorization or delegation”. A valid signature proves that a holder of a key published at a given URL signed the message. It does not prove the agent is well-behaved, that it respects your robots.txt, or that it is entitled to your content. Identity, not permission.

Be honest about the standards status

This is where most write-ups overstate things, so let us be precise about what the IETF datatracker shows as at 26 August 2026.

An IETF working group called webbotauth does exist and is active, in the Web and Internet Transport area, chaired by David Schinazi and Rifaat Shekh-Yusef. Its charter covers cryptographically authenticating automated clients including search crawlers, archivers, AI training crawlers and agents fetching content for a user.

However, the working group has not yet adopted any of the drafts as a working-group document. Every proposal on its document list is still an individual submission — draft-meunier-*, draft-illyes-*, draft-nottingham-*, draft-rescorla-* and others — and none carries the draft-ietf-webbotauth-* name that signals adoption. The datatracker attaches its standard warning to the protocol draft: it “is not endorsed by the IETF and has no formal standing in the IETF standards process”. The intended status is Standards Track, which is an aspiration, not an achievement.

Signs of churn are visible in the detail: Cloudflare’s 1 July 2025 announcement told operators to host their key set “under /.well-known/http-message-signature-directory of your website” — singular — whilst the current draft registers the well-known suffix http-message-signatures-directory, plural. Anyone calling Web Bot Auth “the new standard” is describing where it is heading, not where it is.

How signature verification differs from rDNS and IP ranges

We covered the incumbent techniques in Verifying AI Bot Traffic: rDNS and Published IP Ranges. Both bind a claim to network infrastructure: reverse DNS asks whether the IP resolves to a vendor hostname that resolves back to the same IP, and IP ranges ask whether the address is on the vendor’s list. Both work, and both are awkward — rDNS costs a DNS round trip in the request path, and IP lists must be refetched as they change.

A signature binds the claim to a key instead of an address. That decouples identity from hosting, which matters enormously for user-directed agents that may run from a laptop, a residential connection or an ephemeral cloud sandbox — traffic that can never be captured by a fixed IP list.

Method	Spoofable?	Verification cost	Coverage of AI traffic	Available today?
User-agent string	Trivially — it is free-text	Nil	Everything self-identifies (honestly or not)	Yes, and near worthless alone
Reverse DNS (forward-confirmed)	No, if done forward-confirmed	Two DNS lookups per check; cacheable	Only vendors that set PTR records	Yes, widely
Published IP ranges	No, but lists go stale	Fetch and refresh JSON; fast lookups	Only fixed-infrastructure crawlers	Yes — OpenAI, Google and others publish
Cryptographic signature (Web Bot Auth)	Not without the private key	Key-directory fetch (cacheable) plus per-request signature check	Only agents that have implemented it	Partly — via Cloudflare; spec still pre-adoption

Note the honest trade in the last two columns. Signatures are the strongest guarantee and the thinnest coverage. IP ranges are weaker and far broader. As at August 2026, OpenAI’s own bot documentation still points operators at IP ranges — “we recommend allowing OAI-SearchBot in your site’s robots.txt file and allowing requests from our published IP ranges below” — with no mention of message signatures. Meanwhile the IP-range approach is itself being standardised in the same working group, via Gary Illyes’ draft-illyes-webbotauth-jafar-00, “A JSON-Based Format for Publishing IP Ranges of Automated HTTP Clients”, dated 21 April 2026. These are complementary, not rivals.

What a site operator can do today

If you sit behind Cloudflare, most of this is already done for you. Cloudflare added HTTP Message Signatures to its Verified Bots Program in a 1 July 2025 announcement, and its verified bots documentation now lists two acceptable routes — “Web Bot Auth” or “IP validation” — with honest self-identification demonstrated by “a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS”. On 28 August 2025 Cloudflare introduced a signed agents category for user-directed agents, launching with agents including ChatGPT agent, Goose from Block, Browserbase and Anchor Browser. Its documentation states: “As of July 1, 2026, the distinction between a Verified bot and a signed agent is expressed by a new metadata field tracked in BotBase: Direct versus Intermediary access, which tracks who can operate the bot”. Practically: you write rules against the category, not against strings.

If you don’t, do not rebuild this yourself yet. Three sensible steps:

1. Log the Signature-Agent header. It costs nothing and tells you when signed traffic starts arriving. Most sites are not capturing it, so they have no baseline.
2. Keep doing forward-confirmed rDNS and IP-range checks. They cover the overwhelming majority of AI crawler traffic today and will for some time.
3. Do not block on signature absence. Almost no agent signs yet. Treat a valid signature as a positive signal that earns trust, never treat its absence as evidence of bad faith.

What it means for legitimate AI agents

The interesting half is the other direction. As buying research shifts to agents that browse on a person’s behalf — a shift we wrote about in When the Buyer Is an Agent — those agents need a way to say “I am genuinely acting for a real user” that a defensive site will believe. Without it they get lumped in with scrapers and served a challenge page, and the buyer never reaches your form. Signed identity is the plausible way out, and the draft’s scope note is the right division of labour: the signature establishes who, and the site decides what they may do. Signing is necessary but not sufficient — an agent still has to behave.

A note on where we sit. Zian AI builds autonomous sales agents that work over phone, SMS, email and WhatsApp, so our version of this problem is the telephony equivalent — disclosing clearly that a caller is an AI agent, and getting the brand onto the caller-ID screen. We are not a bot-verification vendor and Web Bot Auth is not something we sell. We track it because the same principle recurs: agents that prove who they are get through, and agents relying on unverifiable self-description eventually get blocked.

Sources & ownership

Claim in this post	Owner	Owner URL (fetched 26 Aug 2026)
Protocol draft name, revision 02 dated 18 August 2026, expiry 19 February 2027, authors, Standards Track intended status, and the “not endorsed by the IETF” boilerplate	IETF Datatracker	https://datatracker.ietf.org/doc/draft-meunier-webbotauth-httpsig-protocol/
No draft-ietf-webbotauth-* document exists; every WG document is still an individual submission	IETF Datatracker	https://datatracker.ietf.org/wg/webbotauth/documents/
webbotauth WG is active in the Web and Internet Transport area; chairs; charter scope	IETF	https://datatracker.ietf.org/wg/webbotauth/about/
RFC 9421 HTTP Message Signatures, February 2024, Standards Track, and its abstract	IETF / RFC Editor	https://www.rfc-editor.org/rfc/rfc9421.html
Signature-Agent header, well-known directory path, media type, required signature parameters, and the scope limitation quote	IETF (Meunier, Major)	https://www.ietf.org/archive/id/draft-meunier-webbotauth-httpsig-protocol-02.txt
IP-range publishing draft: title, version 00, dated 21 April 2026, author Gary Illyes	IETF Datatracker	https://datatracker.ietf.org/doc/draft-illyes-webbotauth-jafar/
Verified Bots validation methods (Web Bot Auth or IP validation); honest self-identification wording; 1 July 2026 Direct/Intermediary metadata field	Cloudflare	https://developers.cloudflare.com/bots/concepts/bot/verified-bots/
1 July 2025 cryptographic verification announcement and the singular /.well-known/http-message-signature-directory path	Cloudflare	https://blog.cloudflare.com/verified-bots-with-cryptography/
Signed agents category, 28 August 2025, and launch participants	Cloudflare	https://blog.cloudflare.com/signed-agents/
Documented stealth crawling: undeclared Chrome-on-macOS user agent, IPs outside the published range, ASN rotation, scale	Cloudflare	https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/
OpenAI still directs operators to robots.txt plus published IP ranges, with no mention of message signatures	OpenAI	https://developers.openai.com/api/docs/bots

FAQ

Is Web Bot Auth an approved internet standard?

No. As at 26 August 2026 it is a set of individual Internet-Drafts. An IETF working group named webbotauth is active in the Web and Internet Transport area, but it has not adopted any draft as a working-group document, so none carries a draft-ietf-webbotauth-* name. The current protocol draft, draft-meunier-webbotauth-httpsig-protocol-02, is intended for Standards Track but the datatracker notes it “is not endorsed by the IETF and has no formal standing in the IETF standards process”.

What underlying standard does it use?

It builds on RFC 9421, HTTP Message Signatures, published by the IETF in February 2024 as a Standards Track RFC. RFC 9421 is a finished standard; Web Bot Auth is the bot-specific profile on top of it, and that profile is still in draft.

Does a valid signature mean I should allow the agent?

No — it answers a narrower question. The draft states explicitly that the protocol “does not authenticate human users, does not provide anonymous authentication, and does not define authorization or delegation”. A signature tells you which key signed the request. Whether that party may crawl, scrape or purchase is your policy decision, expressed in robots.txt and your access rules.

Should I block bots that don’t sign their requests?

Not in 2026. Adoption is early and the vast majority of legitimate AI traffic still identifies via IP ranges and reverse DNS. OpenAI’s bot documentation currently directs operators to its published IP ranges and does not mention message signatures at all. Blocking unsigned traffic today would block nearly every crawler you actually want.

How do I check whether my current bot traffic is genuine?

Compare source IPs against the vendor’s published list, or run forward-confirmed reverse DNS. We walk through both, with commands, in Verifying AI Bot Traffic: rDNS and Published IP Ranges. Expect some hits to fail the check, and read those failures carefully: published IP lists go stale, and legitimate traffic can arrive via an intermediary, so “unverified” is not “proven fake”. That impersonation genuinely happens is documented — Cloudflare’s 4 August 2025 investigation found a crawler using a generic Chrome-on-macOS user agent and “multiple IPs not listed in Perplexity’s official IP range” — but the honest inference from one failed lookup is “investigate”, not “spoofer”.

What should a bot operator do to get verified?

Cloudflare’s 1 July 2025 announcement sets out the sequence: generate a signing key, host a JSON Web Key Set containing the public key at the well-known message-signature directory path, sign requests using a Web Bot Auth library, and register the directory URL through Cloudflare’s Verified Bots form.

Where this lands

User-agent guessing is finished as a trust mechanism; the only question is what replaces it. The answer is not yet a standard — it is an active working group, a fast-moving draft, and one large CDN shipping ahead of the specification. The correct response is cheap: log the header, keep your rDNS and IP checks running, and treat signatures as a bonus signal rather than a gate.

If you are building outbound or inbound AI agents and want to talk about how identity, disclosure and trust play out across web, phone and messaging channels, Apply For Partnership. Zian AI is in waitlist and partnership beta.