Quick answer

Split the log by status code before you read the total. A crawl spike made mostly of 304 Not Modified responses is a crawler re-validating pages it already holds; a spike of 200s on URLs it has never fetched is discovery. RFC 9110 defines the difference precisely, and it changes what you should do next: freshness and new URLs in the first case, nothing at all in the second.

A hit count is two different events added together

Almost every report of AI crawler activity is a single number: requests per bot, per day. That number silently sums two behaviours with opposite meanings — a crawler asking “has this changed since I last took it?”, and a crawler taking a page it has never seen. Added together they look identical; separated by status code they are barely related.

This matters because the hit count is the metric people act on. A doubling gets read as “we are being discovered”, budget follows, and the cause was a scheduled re-validation sweep across an archive indexed months ago. The fix is not a better dashboard. It is one awk pass over the access log before anyone forms an opinion.

What a 304 actually means

The semantics are specified, not vendor-defined. RFC 9110 states that “The 304 (Not Modified) status code indicates that a conditional GET or HEAD request has been received and would have resulted in a 200 (OK) response if it were not for the fact that the condition evaluated to false”, and continues: “In other words, there is no need for the server to transfer a representation of the target resource because the request indicates that the client, which made the request conditional, already has a valid representation …” (RFC 9110, Section 15.4.5).

Read that second clause again. A 304 is the server confirming that the client already has your content: the crawler is not learning anything new, it is refreshing something it stored earlier.

A request becomes conditional when the client attaches a precondition header. If-None-Match makes “the request method conditional on a recipient cache or origin server either not having any current representation of the target resource, when the field value is "*", or having a selected representation with an entity tag that does not match any of those listed in the field value” (Section 13.1.2). If-Modified-Since makes “a GET or HEAD request method conditional on the selected representation’s modification date being more recent than the date provided in the field value” (Section 13.1.3).

The values echoed back came from your earlier responses: an ETag, “an opaque validator for differentiating between multiple representations of the same resource …” (Section 8.8.3), or a Last-Modified “timestamp indicating the date and time at which the origin server believes the selected representation was last modified …” (Section 8.8.2). The caching spec names the exchange: “This process is known as "validating" or "revalidating" the stored response.” (RFC 9111, Section 4.3).

So a 304 is evidence of three things at once: the crawler holds a copy, the copy is still current, and it is maintaining that copy deliberately. A stronger signal of being known than a raw 200 — and a weaker signal of anything changing.

Status class to meaning to action

Status class	What it means when an AI crawler receives it	Action it should trigger
200 OK	Full transfer. The crawler had nothing stored, or its stored copy failed validation. On a URL it has never requested, this is discovery.	Check whether the URL is new. If so, linking and sitemaps are working. If not, something is defeating validation — usually a changing ETag on unchanged content.
304 Not Modified	Re-validation. The crawler holds a valid copy and is refreshing it; no content moved. Volume tracks the crawler’s schedule and your archive size, not your relevance.	Nothing defensive. Your lever is genuine change, not manufactured churn.
301 / 302	The crawler is following an old path. RFC 9110 says a 301 “indicates that the target resource has been assigned a new permanent URI …”.	Fix the source link. Budget spent on redirect chains is budget not spent on real pages, and each hop can drop the request.
403 / 429	You refused. A 403 means “the server understood the request but refuses to fulfill it” (RFC 9110); a 429 “indicates that the user has sent too many requests in a given amount of time ("rate limiting")” (RFC 6585).	Decide whether it was intentional. Deliberate blocks belong in robots.txt; accidental ones from a WAF or rate limiter are a silent visibility leak.
5xx	You broke. To an on-demand fetcher answering a live user question, this is a failed answer in real time, not a retry-later.	Treat as an incident with a timestamp. Alert on 5xx to verified fetchers specifically, not just on overall error rate.

Sources for the quoted definitions: RFC 9110 Section 15.4.2 (301), Section 15.5.4 (403), and RFC 6585 Section 4 (429 — note that 429 is not defined in RFC 9110).

The same 304 means different things to different crawlers

Four kinds of automated client show up in an AI-era log, and the status split reads differently for each. We have covered the underlying split between training bots and on-demand fetchers separately; this is what it looks like at the status-code level.

Training crawlers collect corpus material: OpenAI documents GPTBot as “used to make our generative AI foundation models more useful and safe” (OpenAI, Bots), Anthropic documents ClaudeBot as “collecting web content that could potentially contribute to their training” (Anthropic support). Heavy 304 traffic from these means your pages sit in a set they maintain, not that anything was learned this week.

On-demand fetchers retrieve a page because a user asked something right now. OpenAI’s page describes ChatGPT-User as used “for certain user actions in ChatGPT and Custom GPTs” (OpenAI, Bots). These should show few 304s: they resolve a specific URL once, with a person waiting. A 5xx here is the expensive one.

Search indexers maintain a ranked index; OAI-SearchBot is “used to surface websites in search results in ChatGPT’s search features” (OpenAI, Bots). Their 200:304 ratio is the clearest freshness dial you have: a healthy site converges on mostly-304 with a trickle of 200s on new URLs.

Previewers and link unfurlers fetch once, cache hard, and fire when a human shares a link, so their volume tracks social activity rather than crawling policy. Counting them alongside training crawlers is how “AI traffic” totals get inflated.

Reading a log: the method

These assume the default nginx combined format, where the user agent is the sixth double-quote-delimited field and status and bytes sit in the third. Run them in order. All four only read the log; none of them writes, moves or deletes anything.

Step one — group by user agent and status. Never read a site-wide status distribution; it is dominated by humans.

awk -F'"' '{split($3,f," "); print f[1], $6}' access.log \
  | sort | uniq -c | sort -rn | head -40

Step two — get the 200:304 ratio per agent. This is the number to trend, not the hit count. The columns are 200s, then 304s, then the agent, so an agent that only ever gets 304s still appears.

awk -F'"' '{split($3,f," "); ua=substr($6,1,70); seen[ua]=1;
  if (f[1]==200) a[ua]++; else if (f[1]==304) b[ua]++}
  END {for (u in seen) printf "%7d %7d  %s\n", a[u]+0, b[u]+0, u}' \
  access.log | sort -rn

Step three — separate new URLs from re-fetches. A 200 on a URL the crawler fetched before is a validation failure; a 200 on one it has never touched is discovery. Only the second is good news. Process substitution needs bash or zsh; nothing is written to disk.

comm -13 <(awk -F'"' '$6 ~ /GPTBot/ {split($2,r," "); print r[2]}' access.log.1 | sort -u) \
         <(awk -F'"' '$6 ~ /GPTBot/ {split($2,r," "); print r[2]}' access.log   | sort -u)

Step four — count bytes, not requests. A 304 carries no content, so bytes shipped is the honest measure of how much of your site actually left the building.

awk -F'"' '{split($3,f," "); b[$6]+=f[2]}
  END {for (u in b) printf "%12d  %s\n", b[u], u}' access.log | sort -rn | head

A thousand-request spike that moved almost no bytes is a re-validation sweep. We run this split on our own logs every cycle, and it is routinely the difference between a headline and a non-event.

Verify before you interpret

Everything above assumes the user agent is telling the truth. It is a request header: anyone can set it, and scanners routinely do, which is why an unverified “AI crawler surge” often is not one. Identity comes from the network, not the string — the mechanics are in our note on verifying AI bot traffic with reverse DNS and published IP ranges.

Cloudflare frames the same requirement from the operator’s side. The first of its two bars for a verified bot is “Honest self-identification — it declares who it is deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS” (Cloudflare, Verified bots).

Operator	Verification method the operator itself documents	Published range list
Google	Reverse DNS to googlebot.com, google.com or googleusercontent.com, then a forward lookup back to the same IP; or CIDR matching	Yes — separate files for common crawlers, special-case crawlers and user-triggered fetchers (verification page, common-crawlers.json)
Apple	“Traffic coming from Applebot is generally identified by using reverse DNS in the *.applebot.apple.com domain”, or CIDR matching	Yes — applebot.json (About Applebot)
OpenAI	IP range matching. The bots page gives a range file per user agent and describes no reverse-DNS method	Yes — one file per agent, e.g. gptbot.json (OpenAI Bots)
Anthropic	IP range matching: “If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic”	Yes — bots.json (Anthropic support)
ByteDance (Bytespider)	We could not locate a crawler verification page or range list on ByteDance’s own properties	Not published, as far as we can find. Treat matching hits as unverified rather than attributing them

All four published files share a shape: a creationTime and a list of ipv4Prefix / ipv6Prefix entries. Match the source IP against those prefixes and you have an identity; fail to, and you have a string. Report an unverifiable agent as its own line item labelled unverified, never folded into a total that implies it was checked.

What each pattern should change

Mostly 304, stable URL set. Your content is known and considered current: nothing to fix, nothing to celebrate. Your only lever is genuine change — new URLs, and real updates that legitimately move Last-Modified. Re-saving pages to bump timestamps teaches a crawler that your dates are noise.

Mostly 200 on first-seen URLs. Discovery is working. Confirm they are the URLs you wanted crawled and check what linked them — this is the one pattern where a rising hit count means what people assume it means.

Mostly 200 on URLs already fetched. Something is defeating validation — usually an ETag that changes on every response, a missing Last-Modified, or a CDN rewriting validators. You pay bandwidth for re-transfers of unchanged pages, and the crawler spends budget it could have spent on new content.

4xx or 5xx to a verified fetcher. An incident. If a fetcher resolving a URL for a user mid-conversation got a 500, that answer was degraded at that moment. If the 403s are yours by policy, they should be a deliberate allowlist decision rather than a WAF rule nobody remembers writing.

What not to conclude

Crawl volume is not citation. No crawler sends a request meaning “we used your page in an answer”, and we could find no operator publishing a per-URL citation feed to publishers. A fetch tells you the page was retrievable; whether it was ever surfaced to a person is measured elsewhere, which is why we keep crawl logs and answer-surface measurement in separate columns and treat being cited and being recommended as distinct outcomes.

Two more traps: a 200 is not proof of indexing, because fetching and storing are separate decisions made at different times; and byte volume is not value, because a crawler that pulls your whole archive once has told you nothing about whether any of it was useful.

FAQ

Is a high 304 count from an AI crawler good or bad?

Neither by itself. It means the crawler holds copies of your pages and is checking whether they changed. Per RFC 9110, a 304 is only sent because “the client, which made the request conditional, already has a valid representation” (RFC 9110, Section 15.4.5). It is good in the sense that you are in the set being maintained, and uninformative as a growth metric.

Should I remove ETag or Last-Modified so crawlers re-fetch everything?

No. You would force full transfers of unchanged pages, spend the crawler’s budget on content it already has, and lose your only signal for which pages genuinely changed. Serve accurate validators and let the 304s happen.

How do I tell a real GPTBot from something spoofing it?

Match the source IP against the range file OpenAI publishes for that specific agent — openai.com/gptbot.json for GPTBot, with separate files for OAI-SearchBot and ChatGPT-User (OpenAI, Bots documentation). The user agent alone proves nothing.

Does a crawl spike mean my content is being used in AI answers?

No, and there is no request that would tell you so. Crawling establishes retrievability. Whether an assistant surfaced you in a specific answer has to be measured against the answer surface itself, on the prompts you actually care about.

What about crawlers that publish no IP ranges at all?

Report them as unverified and keep them out of any total you present as verified traffic. One of Cloudflare’s two bars for a verified bot is that it “declares who it is deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user-agent, or reverse DNS” (Cloudflare, Verified bots). If none of those three exist for an agent, no amount of log analysis will establish its identity.

Reading logs properly is the unglamorous half of AI visibility: it stops you acting on a number that was never what you thought it was. The same discipline runs through how we build sales agents at Zian AI — measure the outcome, not the activity that resembles it. If that is how your team wants to work, Apply For Partnership.