How reliable is it

Read this before you quote a number from the AI panel to anyone.

There are three separate reliability questions, and they have different answers. The first is about coverage — how much crawler traffic reaches us at all. The second is about identity — whether a crawler that did reach us is who it says it is. The third is about the AI traffic that never looks like AI traffic at all.

1. Coverage: the browser script alone undercounts

Most AI crawlers request raw HTML and never run JavaScript. A browser tracking script cannot see them. Server-side tracking is what closes this gap, and everything in this section describes a site that has not added it yet.

That is not a design choice or a bug. It is what a JavaScript tracker is. GPTBot fetches your page's HTML, reads it, and leaves. Our script — which only runs when a browser executes it — is never executed, so no event is ever sent.

What this means in practice

So the panel's number is a floor, not a total. Every crawler view it shows is real. There are more that it does not show, and we cannot tell you how many.

How much is missing depends on your site

This is the part that determines whether the current number is useful to you.

If your pages are server-rendered and your server also serves our script, the gap is smaller — a crawler fetching a rendered page still makes a request that can be recorded.

If your site is a static marketing site or a client-rendered app, the gap is larger. A crawler fetching your static HTML from a CDN produces nothing that reaches us at all.

We are not going to put a percentage on this, because we would be making it up.

What closes the gap

Server-side reporting is the answer, and it is available now: a small hook on your own server that reports crawler requests directly, including every one that skips JavaScript.

Two things worth knowing before you add it:

• It is optional. The zero-install browser path stays the default. The server-side path is for people who want complete coverage, not a prerequisite for using the panel at all.

• Your numbers jump the day you switch it on. That is the missing traffic becoming visible, not a change in your crawler traffic. Do not read the step change as growth.

Without it, read the panel relatively rather than absolutely: which operators are reading you, which of your pages they prefer, and whether that is trending up or down. Those comparisons hold even under a constant undercount. The absolute count does not.

2. Identity: a crawler is identified by what it claims

A crawler is classified from its user-agent string — a header the client chooses. Anything can send any user agent. A scraper can call itself GPTBot and we will initially record it as GPTBot.

That is true of every product that reports crawler traffic, including ours. Here is what we do about it.

The verification tiers

Every crawler view carries one of these:

Spoofed is surfaced, never hidden. A crawler impersonating OpenAI on your site is something you should be able to see.

The difference between the last two matters and is easy to blur: unverified is an absence of proof, and spoofed is proof of the opposite. A published address list can be incomplete or behind, so a miss is not evidence that anyone is pretending.

Which operators can be verified at all

Thirteen operators, by one of two methods: some publish their address ranges, and others publish reverse DNS records instead — Yandex, Baidu and Huawei are in the second group, which is why they verify despite publishing no list. Which crawlers names the method for each one.

The right-hand column is not a limitation of ours. Meta, ByteDance and xAI publish nothing at all, and PerplexityBot's list has not moved since February 2025 — checking against it would mark real visits as spoofed, which is worse than saying nothing.

Perplexity is the split case worth knowing: its Perplexity-User list is thin but current and is used, while PerplexityBot stays unverified. So one of the two verifies and the other cannot.

For the right-hand column, "unverified" is the honest ceiling. A Meta hit can be reported and never confirmed — and equally, the absence of a confirmed Meta hit is not evidence that Meta did not visit.

What the share actually looks like

Verification runs on every crawler visit that arrives with an address, so the "Verified crawlers" tile is a real measurement rather than an aspiration.

Across all sites over the seven days to 17 September 2026: 44.6% verified, 55.3% unverified, 0.1% spoofed. A number in the forties is what a healthy site looks like, not a failure — the unverified majority is mostly operators who publish nothing to check against, plus visits that reached us without an address.

Two things follow, and both are worth knowing before you read the tile:

• A low share is not a problem to fix. It is mostly a fact about which operators crawl you. A site read mainly by ByteDance sits lower than one read mainly by OpenAI, and neither is doing anything wrong.

• Rows collected before verification existed stay unverified. They cannot be checked after the fact, because the address they came from was never stored.

3. Some AI traffic is invisible by design

The two limits above both have a fix you can apply: server-side reporting closes the coverage gap, and address checking closes the identity gap. This third one has none, and you should know about it before you treat any AI number as complete — ours or anyone else's.

An AI agent driving a real browser looks exactly like a person.

Agentic browsers — ChatGPT Atlas, Perplexity Comet, Claude for Chrome — do not send a crawler user agent. They drive an actual browser session, so what arrives is Chrome's user agent from a residential address. There is nothing in the request that says an AI is at the controls.

The same is true of AI features built into browsers you already have, and of coding agents that fetch a page through a normal browser rather than announcing themselves.

What that means for your numbers

• AI answers is a floor, not a total. Some assistant traffic is counted as ordinary human visits, and we cannot separate it.

• The gap grows as agentic browsing grows. This is the direction the industry is moving, so expect this limit to matter more over time, not less.

• No user-agent-based product can fix this, including ours. A number derived from user agents can only see clients that identify themselves.

We would rather you knew this than found out later from a number that felt wrong. Where AI traffic identifies itself, we report it and say how confident we are; where it does not, we say so here rather than quietly rounding it into the human column and letting you believe the AI figure is the whole picture.

A published list can still be incomplete

Anthropic's own documentation says IP-based identification "may not work correctly or permanently" — the list is a help, not a guarantee.

We measured this. A genuine Claude fetch reached a site from 23.251.146.115, and that address is not in claude.com/crawling/bots.json. The list holds 26 ranges, mostly Google Cloud; this address is Google Cloud too, and still absent.

So "unverified" does not mean "not Claude". It means we found no proof either way. This is why a missing address never produces a spoofed stamp: being off a list that the operator admits is incomplete is not evidence of anything. Only reverse DNS, where the address demonstrably belongs to somebody else, can conclude that.

One operator we cannot see at all

xAI (Grok) publishes no crawler user agent and no address ranges. Other products list tokens like GrokBot or xAI-Grok; we checked xAI's own sites and found no such string documented anywhere, and the third-party lists carrying them contradict each other.

We are not adding an invented token to make the panel look complete. A row that never matches anything reads as "Grok is not reading your site", which is a different claim from "we cannot tell", and only one of those is true.

What this panel is good for today

• Which operators are reading you, relatively.

• Which of your pages they prefer — see AI traffic by page.

• The assistant-versus-training split, which is a different and more useful question than "how many bots".

• Trends over time, which survive a constant undercount.

What it is not good for today

• An absolute count of AI crawler visits, on a site reporting from the browser alone. It is a floor until you add server-side tracking; with it, every crawler that reaches your server is recorded.

• Proving a crawler did not visit. Absence here is not evidence of absence, especially for Perplexity, Meta and ByteDance.

• Confirming a crawler was genuine, for every operator. The thirteen listed above verify; the rest publish nothing to check against, and for those an unverified claim is the ceiling.

If any of that changes, this page changes with it. A claim on this page that stopped matching the product would be a fault, not a documentation backlog item.