If you tally “AI bot traffic” by grepping your access log for GPTBot or ClaudeBot, your numbers are wrong. Anyone can send any user-agent header — and vulnerability scanners, scrapers and SEO tools increasingly borrow AI-crawler identities precisely because so many sites now allowlist them. The result: inflated bot counts, spoofed hits treated as evidence that AI engines are reading your content, and firewall rules that let hostile traffic through.

The fix is a verification workflow that vendors themselves publish. This post covers the two mechanisms on offer, which vendors support which, and a worked command-line workflow you can run against your own logs today. If you’re still sorting out which AI crawlers you want visiting in the first place, start with our guide to training bots versus on-demand fetchers — the verification step below applies to both kinds.

Step 1: Treat the user-agent string as a claim, not proof

An HTTP request has exactly one field the client fully controls and nobody validates: the User-Agent header. When a request arrives claiming to be GPTBot/1.4, all you actually know is that someone typed that string. Common Crawl puts it plainly on its own crawler page: “we are aware of crawlers falsely identifying themselves as CCBot” — and the same impersonation happens to every well-known AI bot identity.

So the first rule: never make an access decision, and never report a traffic number, on the user-agent alone. The user-agent tells you which verification method to apply next; it proves nothing by itself.

Step 2: Reverse DNS with forward confirmation (where supported)

The strongest self-serve check is forward-confirmed reverse DNS (FCrDNS). It’s two lookups, not one:

1. Reverse: look up the PTR record for the connecting IP. It must resolve to a hostname under the vendor’s verification domain.
2. Forward: resolve that hostname back to an IP. It must match the original connecting IP.

The forward step matters because anyone who controls the reverse zone for their own IP space can set a PTR record saying fake.googlebot.com.evil.example. Only the round trip proves control of the vendor’s real domain. Here it is against a genuine Googlebot address (Google’s documented verification domains are googlebot.com, google.com and googleusercontent.com):

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1   # matches — verified

We ran the same check live against Apple’s and Common Crawl’s documented examples while writing this post: 17.58.101.179 resolves to 17-58-101-179.applebot.apple.com and 18.97.14.84 to 18-97-14-84.crawl.commoncrawl.org, each forward-confirming correctly. By contrast, an IP inside OpenAI’s published GPTBot range returned no PTR record at all — which is expected, because OpenAI doesn’t offer rDNS verification. That’s not a red flag; it just means you use their other mechanism.

Step 3: Published IP-range JSON files

Most AI vendors have converged on a simpler mechanism: a machine-readable JSON file listing the exact CIDR ranges their bots egress from, usually in the same prefixes format Google pioneered. If the connecting IP falls inside a published range, the request is genuinely from that vendor; if the user-agent claims a bot but the IP is outside every published range, it’s a spoofer.

Here’s the current state of play. Every endpoint below was fetched on 22 August 2026 and returned HTTP 200 with live range data.

*A note on Google-Extended, because it’s widely misunderstood: it is a robots.txt control token, not a separate crawler. Google’s crawler documentation states that Google-Extended “doesn’t have a separate HTTP request user agent string” — the fetching is done by Google’s ordinary crawl infrastructure, so you verify it exactly as you verify Googlebot. You will never see “Google-Extended” in a legitimate user-agent header; if you do, it’s a spoofer by definition.

Three practical details. First, these files change: OpenAI’s ChatGPT-User list carried a creation timestamp of 14 August 2026 when we fetched it — just over a week old, and Google regenerates its files continually — so re-fetch on a schedule (daily is plenty) rather than hard-coding ranges. Second, Anthropic’s support article explicitly warns that IP blocking “may not work correctly or persistently guarantee an opt-out” — the JSON is for verifying inbound traffic, while robots.txt remains the supported opt-out channel. Third, some vendors publish only aggregate files (Anthropic covers all three of its bots in one list), so the JSON tells you the request is genuinely theirs, and the user-agent then tells you which product sent it.

Apply For Partnership

Step 4: A worked workflow — cross-checking your log against the ranges

Here’s a minimal pipeline for an nginx access log. Pull every IP that claimed an OpenAI identity, then test each against the published ranges using Python’s standard library (no dependencies):

# 1. Collect claimed-GPTBot IPs from the log
awk '/GPTBot|OAI-SearchBot|ChatGPT-User/ {print $1}' /var/log/nginx/access.log | sort -u > claimed.txt

# 2. Fetch the vendor's current ranges
curl -s https://openai.com/gptbot.json \
     https://openai.com/searchbot.json \
     https://openai.com/chatgpt-user.json > ranges.json

# 3. Verify each IP against the ranges
python3 - <<'EOF'
import ipaddress, json, re
nets = [ipaddress.ip_network(v) for v in
        re.findall(r'"ipv[46]Prefix":\s*"([^"]+)"', open('ranges.json').read())]
for line in open('claimed.txt'):
    ip = ipaddress.ip_address(line.strip())
    print(ip, "VERIFIED" if any(ip in n for n in nets) else "SPOOFED")
EOF

For rDNS vendors, the equivalent one-liner per IP is host $IP followed by host on the returned name, checking the suffix and the round trip as shown in step 2. Run whichever check the vendor supports; where both exist (Google, Microsoft, Apple, Common Crawl), either suffices — Common Crawl’s own JSON file goes as far as recommending you do both for IPv4.

Expect a gap. Impersonating well-known bot identities is a standard scanner tactic — Common Crawl warns about it happening to CCBot on its own crawler page — so a UA-only count will overstate genuine AI activity by whatever your spoof rate happens to be. Measure that rate before you quote any bot numbers internally, and reconcile the verified picture against what actually shows up in analytics, per our guide to measuring AI referral traffic in GA4.

Step 5: What to do with each verdict

Confirmed spoofers: rate-limit or block, quietly

A request that claims a bot identity and fails IP verification has already told you it’s willing to lie. Sensible responses, in escalating order: exclude it from all bot reporting; rate-limit the source IP; return 403 on the impersonated user-agent from unverified ranges. In nginx, a geo block listing the vendor’s published CIDRs combined with a user-agent map lets you 403 exactly the mismatch — the claimed identity without the verified source — while touching nothing else.

Verified bots: a deliberate allowlist decision, not a default

Verification tells you the request is genuine; it doesn’t tell you whether to welcome it. A verified GPTBot hit is a training crawl; a verified ChatGPT-User or Claude-User hit is a live human asking an assistant about you right now; a verified CCBot hit feeds an open dataset. Those deserve different answers depending on your strategy — which is a robots.txt policy question, and we’ve laid out a full decision framework in our AI crawler allowlist guide for B2B SaaS. The point of verification is that whatever policy you choose, it’s now being applied to the bots you think it is.

One more operational note: keep your verification data fresh and your logs honest before AI visibility becomes commercially important to you — because once AI assistants are answering buyer questions with your content, you’ll want to trust every number in that report. That’s the same discipline we apply at Zian AI to the other side of the equation: our digital sales agents (from the Outbound Appointment Setter to the 24/7 Customer Support Agent) operate on verified, measured outcomes rather than vanity counts, with SmartReach AI™ deciding message, channel and timing from real signals.

Sources & ownership

Frequently asked questions

Why isn’t the user-agent string enough to identify an AI bot?

Because the client sets it freely and nothing validates it. Scanners and scrapers deliberately impersonate well-known AI crawlers to slip through allowlists, so a user-agent match only tells you which verification check to run next — reverse DNS or the vendor’s published IP ranges.

Which AI vendors support reverse-DNS verification?

Google (hostnames under googlebot.com, google.com or googleusercontent.com), Microsoft Bingbot (*.search.msn.com), Apple (*.applebot.apple.com) and Common Crawl (*.crawl.commoncrawl.org) support forward-confirmed reverse DNS. OpenAI, Anthropic and Perplexity do not offer rDNS; for them you must check the connecting IP against their published range files.

Where do I find each vendor’s official IP-range file?

OpenAI serves per-bot files at openai.com (gptbot.json, searchbot.json, chatgpt-user.json); Anthropic serves one combined file at claude.com/crawling/bots.json; Perplexity at perplexity.ai/perplexitybot.json and perplexity-user.json; Google at developers.google.com under /static/crawling/ipranges/; Microsoft at bing.com/toolbox/bingbot.json; Apple at search.developer.apple.com/applebot.json; Common Crawl at index.commoncrawl.org/ccbot.json. All were live and returning current range data when this post was published.

Is one verification method enough, or should I run both?

Either passing check is strong evidence on its own, but Common Crawl’s own published range file at index.commoncrawl.org/ccbot.json — fetched and confirmed live for this post — states in its notes field: “For verification of IPv4 addresses, FCrDNS is also recommended.” Where a vendor offers both mechanisms, running both costs one extra DNS lookup and removes any doubt.

Should I block a bot that fails verification?

Failing verification means the request lied about its identity, so at minimum exclude it from bot reporting; rate-limiting or a 403 on the spoofed user-agent from unverified IPs is reasonable. Genuine verified bots are a separate, deliberate policy decision — a training crawler, a user-triggered fetcher and an open-dataset crawler each warrant their own robots.txt answer.

Apply For Partnership