Which crawlers are recognised

The current list. It is extended as operators launch new crawlers; nothing is removed, so a name that appeared in your history stays readable.

By operator

OpenAI

OAI-AdsBot checks pages submitted as ChatGPT ads. It is not answering a question and not collecting training data, so it is counted as an OpenAI visit without being placed in either AI category.

OAI-SearchBot and Claude-SearchBot count as indexing, not as AI answers. Both operators describe them as the crawler that builds their search index — the fetch that happens so your page can be found later, not the one that happens while somebody is being answered. They were counted as AI answers until September 2026; if your AI answer figure dropped and your indexing figure rose on the same day, this is why.

Anthropic

Perplexity

PerplexityBot is indexing, not training. Perplexity's own documentation says it builds a search index and does not train models on what it fetches. The distinction matters when you write robots.txt: blocking training and keeping search visibility are different decisions.

The two bots differ on verification, because Perplexity publishes two lists in very different states. The file covering Perplexity-User was updated in October 2025 and is used. The one covering PerplexityBot has not moved since February 2025 and holds eight generic cloud addresses — checking against a list that old produces false assurance rather than verification, so we refuse it and a PerplexityBot hit is reported without ever being confirmed. See How reliable is it.

Google

Googlebot and GoogleOther are separated deliberately. Googlebot indexes for search; GoogleOther collects for AI and research. Reporting them as one "Google" number would be the single most misleading thing this panel could do, since it would let ordinary search indexing masquerade as AI interest.

Microsoft Bing

Copilot is matched only where it appears as a crawler token — with a version after it, or at the end of the string. The word also occurs inside user agents that are not this crawler, and matching those would file a developer's editor traffic as a Microsoft AI crawler.

Apple

Applebot serves Siri and Spotlight Suggestions.

Meta

Meta's three bots do three different jobs, so they are counted separately. meta-externalagent is one of the highest-volume crawlers on the web; grouping it with the other two would swamp them.

Meta publishes no IP ranges, so its traffic is permanently unverifiable.

ByteDance

Also publishes nothing. Bytespider does not honour robots.txt.

TikTokSpider is listed as training here, but sources disagree — some describe it as a link-preview fetcher rather than a training crawler. We count it on the AI panel because that is the more cautious reading: showing a visit that turns out to be a preview bot is a smaller error than hiding a training crawler.

Mistral

Mistral splits its crawling into exactly these three jobs and documents each one. MistralAI-Training is the one without a published address list.

Amazon

Amazon shows its crawler addresses on a web page rather than in a machine-readable file, and publishes no reverse DNS record either. There is nothing to check an address against, so an Amazon visit is reported and never confirmed.

Common Crawl

Common Crawl trains no models itself, but its corpus is a training input for most of the models above — which is why it is on the AI panel rather than filed as a generic scraper.

Moonshot AI

Like Mistral, Moonshot splits its crawling into three documented jobs and publishes an address list for each.

DuckDuckGo

DuckDuckGo keeps the most current address list of any operator here.

Alibaba, xAI, Cohere, DeepSeek, Zhipu and Allen AI

These operators are named on your dashboard rather than lumped into "Other", but none of them publishes an address list or a reverse DNS record — so the honest ceiling for all of them is an unverified claim.

xAI is the thinnest case. It publishes no crawler documentation and does not appear in the community-maintained agent lists either. The tokens below are recognised from observed traffic, several of which sign themselves with an x.ai URL:

A bare Grok is deliberately not matched: one word fires on any user agent that merely contains it, and a browser extension would be reported as xAI traffic.

Other named AI crawlers

Reported under Other, grouped by what they are doing.

AI training — collecting text for a model: Diffbot, ImagesiftBot, omgili, Timpibot, LAIONDownloader, ICC-Crawler, Cotoyogi, Webzio-Extended, img2dataset.

AI training via a scraping service — these fetch your page on behalf of something else, so the model behind them is not identifiable: FirecrawlAgent, ApifyBot, Brightbot, Crawl4AI, Crawlspace, Lightpanda, CragCrawler.

AI assistant — fetching because someone asked a question: kagi-fetcher, iaskspider, iAskBot, WRTNBot, ExaBot, ExaSearchBot, TavilyBot, LinkupBot, AddSearchBot, Cloudflare-AutoRAG, AzureAI-SearchBot, Channel3Bot, TerraCotta, HenkBot, QueritBot.

Coding and browsing agents — also assistants, since a person is waiting: Devin, Cline, OpenHands, Cursor, Trae, opencode, Manus-User, PhindBot, LinerBot, Andibot.

Most operators in this section publish no address ranges, so their visits stay unverifiable.

Other search engines

Reported under Other, as search indexing: Slurp (Yahoo), SeznamBot, Qwantify, Bravebot (Brave), MojeekBot, Yeti (Naver).

Yandex, Baidu, Huawei, You.com and DuckDuckGo have their own rows on your dashboard rather than appearing here — YandexBot, Baiduspider, PetalBot, YouBot, ERNIEBot, YiyanBot, PanguBot and DuckDuckBot are named by operator. Yandex, Baidu and Huawei publish reverse DNS records, so their visits can be verified even though they publish no address list.

Social and link preview fetchers

facebookexternalhit, Twitterbot, LinkedInBot, Slackbot, Slack-ImgProxy, Discordbot, WhatsApp, TelegramBot, Pinterestbot, redditbot, SkypeUriPreview, vkShare, Iframely, Embedly — reported under Other bots, not as AI. These fetch your page to build a link preview when someone shares it. Useful to see, but not AI traffic.

Keeping them out matters more than it sounds: every share triggers one, so a popular post can produce more preview fetches than you have real AI visits. Some public crawler lists file facebookexternalhit as a possible AI crawler; we do not, because counting it would visibly inflate your AI number.

Anything else that identifies as a bot

A user agent containing bot, crawler, spider, scraper, HeadlessChrome, PhantomJS, curl/ or wget/ is recorded as Other bots.

Crawlers we cannot recognise

Two gaps worth naming, since neither shows up as a missing row — they show up as a number that is quietly too low.

A crawler that sends a name nobody has published. New operators appear faster than any list is updated, and one whose token we do not carry is counted as a generic bot rather than by name. That understates the AI figure rather than inflating it, which is the right direction to be wrong — but it is a real gap, and the fix is a report: send the user-agent string and it gets added.

Agentic browsers send a normal browser's user agent. ChatGPT Atlas, Perplexity Comet and Claude for Chrome drive a real browser, so their visits arrive looking like an ordinary person on Chrome. No user-agent-based product can separate them, ours included. See How reliable is it.

Google-Extended and Applebot-Extended

Neither will ever appear on this panel, and their absence is not a gap in our coverage.

They are not crawlers. They are robots.txt control tokens. Google's own documentation states that Google-Extended "doesn't have a separate HTTP request user agent string" — it governs whether content Google has already crawled may be used to train Gemini. Applebot-Extended is the same for Apple's models.

Neither one makes a single request. There is nothing to count.

So if you have blocked Google-Extended in your robots.txt and are checking whether that worked, this panel cannot tell you — nothing would have appeared either way. What you would look at instead is GoogleOther, which is a real crawler that does make requests.

Any row claiming to be Google-Extended would be either a spoofer or a bug in our classifier, which is why the database refuses to store the value at all rather than trusting the classifier to keep getting it right.

When something is missed

A crawler we do not recognise is counted as a human.

That direction is deliberate. It understates the AI panel rather than inflating it, and for a feature sold on honesty, understating is the correct way to be wrong.

It does mean an unrecognised crawler counts against your plan's events rather than against the crawler allowance, which is the one cost of erring this way. The alternative — guessing that unfamiliar traffic is a bot — would move real visitors off your numbers, and a visitor you never knew you had is a worse error than an event.

If you are seeing traffic you believe is a crawler and it is not showing here, send the user-agent string to support@truestat.io and it can be added.