Recording crawlers from your server

The tracking script runs in a browser. Most AI crawlers do not use one — they request your HTML and never execute JavaScript, so the script never loads and the visit is never recorded.

On a real site the difference is large. Before server-side tracking was added, one site in our own account had recorded 4 crawler visits against 203 human ones over three days: two Bingbot hits and two test requests. No GPTBot, no ClaudeBot, no Googlebot, on a site those crawlers visit constantly.

If the AI panel looks emptier than you expect, this is why.

What you send

One HTTP request from your server, per crawler visit or in batches:

POST https://truestat.io/api/crawl
Content-Type: application/json

{
  "site_key": "ts_your_site_key",
  "hits": [
    {
      "user_agent": "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)",
      "path": "/pricing",
      "ip": "20.171.207.14",
      "status": 200
    }
  ]
}

Only site_key, user_agent and path are required. For a single visit you can skip the hits array and put the fields at the top level.

Up to 50 hits per request.

The response tells you if it worked

{ "ok": true, "recorded": 3, "skipped": 12, "verified": 1, "spoofed": 0, "paused": false }

• recorded — crawler visits stored

• skipped — requests that were not crawlers, ignored rather than rejected

• verified — confirmed as genuinely from that operator

• spoofed — the user agent claimed an operator the IP does not belong to

• paused — false until your crawler allowance for the period is used up

Crawler events have an allowance of their own, separate from the events your plan counts, so a crawl-heavy month never eats into your human traffic. When that allowance is used up the call still answers 202, but records nothing:

{ "ok": true, "recorded": 0, "skipped": 3, "verified": 0, "spoofed": 0, "paused": true, "reason": "bot_quota_exhausted" }

Your tag and your human visits are unaffected. Recording resumes when the period renews, or sooner with a crawler pack — see What counts as an event.

skipped is the useful one during setup. If you send every request and see recorded: 0, skipped: 40, no crawler reached you in that window. If you see recorded: 0, skipped: 0, nothing arrived at all and the call is not firing.

Human traffic is ignored, not rejected

Send every request if that is simpler. Anything that is not a crawler is skipped rather than stored, so your human visits are never counted twice — the script already records those.

Verification

Passing ip lets us check the crawler is who it says it is. A user agent is a claim: anyone can run curl -A "GPTBot", and without checking, a scraper inflates your AI report.

The check is forward-confirmed reverse DNS, the method Google and Bing document. The IP's reverse record must belong to the operator, and that name must resolve forward to the same IP. The second step matters — a reverse record alone can be set by whoever controls the address.

Three outcomes:

Two sources are checked. Where the operator publishes its address ranges — OpenAI, Anthropic, Google, Apple, Mistral, DuckDuckGo, Moonshot, Common Crawl and Perplexity's user fetcher — the address is looked up in that list first. Where the operator documents a reverse-DNS suffix instead — Google, Bing, OpenAI, Apple, Yandex, Baidu and Huawei — the two-way DNS check runs.

Unverified is not suspicion. Several operators publish nothing to check against, so their visits can never be more than unverified:

Perplexity is the split case. Its list for Perplexity-User is thin but is Perplexity's own, so it is used. Its list for PerplexityBot has not been updated since 2025 and holds a handful of generic cloud addresses — checking against it would mark real visits as spoofed, which is worse than saying nothing, so that bot stays unverified. Meta, ByteDance and xAI publish nothing at all.

Sending ip is optional. Without it every visit is recorded as unverified.

Installing it

Every example does the same three things: decide whether the request is worth reporting, send it without making the visitor wait, and never let a failure here reach your page.

If you serve the script from your own domain

Proxying moves the collector to a hostname of yours, and that applies to this call as well as to the browser tag — same reason, same gain: a request to a.example.com is not on the lists ad blockers filter.

Point the endpoint at your hostname. With the package it is one option:

export const proxy = truestat("YOUR_SITE_KEY", {
  endpoint: "https://a.example.com",
});

Writing the call by hand, it is the URL you post to. Your install screen fills this in for you once the managed proxy is verified — the note under the tag says which hostname it loads from, and the crawler call beneath it uses the same one — so the snippet there is always the right one. That is why re-copying it after you turn the proxy on is worth the minute.

Leave it pointing at truestat.io and nothing breaks; that endpoint stays valid. You simply keep the blockable hostname for this half of the install.

How each one decides what to send

Two loose checks on the user agent, not a list of crawler names:

const MAYBE_BOT = /bot|crawl|spider|\+http|compatible;|headless|preview/i;
const LOOKS_LIKE_BROWSER =
  /Mozilla\/5\.0 .*(Chrome|CriOS|Firefox|FxiOS|Safari|Edg|OPR|SamsungBrowser)\//;

// Send if: it says it is a bot, or it does not look like a browser.
MAYBE_BOT.test(ua) || !LOOKS_LIKE_BROWSER.test(ua)

A name list would go stale every time an operator launches something new, and this code lives on your server rather than ours — we cannot update it for you. Checking the shape instead catches crawlers that do not exist yet.

A real visitor on Chrome matches the second check and is skipped, so you are not paying for an outbound request on every human pageview. A request with no user agent at all is sent: nothing without one is a browser.

The package adds a third check, on the path. Requests under /api/, /_next/, /_nuxt/, /_astro/, /static/, /assets/, /cdn-cgi/ and /.well-known/ are never reported, and neither is anything with a file extension — otherwise every image a crawler loads becomes its own record. The files crawlers come for are the exception and are always reported: robots.txt, llms.txt, sitemaps, feeds and Markdown. ignorePaths adds prefixes of your own. The hand-written versions below skip nothing by path; add the same rule if your server hands out many files.

Next.js

On Node, install the package rather than pasting code.

npm install @truestat/ai-crawl

// proxy.ts — middleware.ts on Next 15 and earlier
import { truestat } from "@truestat/ai-crawl/next";

export const proxy = truestat("YOUR_SITE_KEY");

export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

Already have a middleware? Call it from yours:

const report = truestat("YOUR_SITE_KEY");

export function proxy(request: NextRequest, event: NextFetchEvent) {
  report(request, event);
  return NextResponse.next();
}

It does everything the hand-written version below does, and two things besides: it reads the visitor's IP from whichever header your platform sets — Cloudflare, Vercel and Fly each use a different one — and it reports through waitUntil, so the work is not cancelled when the response is returned.

Or write it yourself

The same thing in full, for reading — or for a project that would rather not add a dependency.

// Next.js 16+: proxy.ts  ·  15 and earlier: middleware.ts
// Rename the function to match the file.

// Should this request be reported at all? A crawler either says so, or
// does not look like a browser. Both checks are loose on purpose —
// TrueStat classifies properly on its side, and a wasted request costs
// far less than a crawler that goes unrecorded.
//
// This is NOT about which browser a visitor uses; that is worked out
// from the user agent on TrueStat's side and shown in your dashboard.
// Here the only question is whether to send the request.
const MAYBE_BOT = /bot|crawl|spider|\+http|compatible;|headless|preview/i;
const LOOKS_LIKE_BROWSER =
  /Mozilla\/5\.0 .*(Chrome|CriOS|Firefox|FxiOS|Safari|Edg|OPR|SamsungBrowser)\//;

export async function proxy(request: Request) {
  const ua = request.headers.get("user-agent") ?? "";

  if (MAYBE_BOT.test(ua) || !LOOKS_LIKE_BROWSER.test(ua)) {
    // Fire and forget — never make a visitor wait on analytics.
    void fetch("https://truestat.io/api/crawl", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        site_key: "ts_your_site_key",
        user_agent: ua,
        path: new URL(request.url).pathname,
        ip: request.headers.get("x-forwarded-for") ?? "",
      }),
    }).catch(() => {});
  }
}

export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

Express · Node

npm install @truestat/ai-crawl

import { truestatMiddleware } from "@truestat/ai-crawl/node";

app.use(truestatMiddleware("YOUR_SITE_KEY"));

Mount it before your routes. This one also reports the status code, because an Express middleware can wait for the response to finish — a crawler hitting a 404 is information too.

Or write it yourself

Mount before your routes. Also covers Fastify, Koa, Hapi and NestJS with the equivalent hook.

// Mount before your routes, in server.js or app.js.

// Should this request be reported at all? A crawler either says so, or
// does not look like a browser. Both checks are loose on purpose —
// TrueStat classifies properly on its side, and a wasted request costs
// far less than a crawler that goes unrecorded.
//
// This is NOT about which browser a visitor uses; that is worked out
// from the user agent on TrueStat's side and shown in your dashboard.
// Here the only question is whether to send the request.
const MAYBE_BOT = /bot|crawl|spider|\+http|compatible;|headless|preview/i;
const LOOKS_LIKE_BROWSER =
  /Mozilla\/5\.0 .*(Chrome|CriOS|Firefox|FxiOS|Safari|Edg|OPR|SamsungBrowser)\//;

app.use((req, res, next) => {
  next(); // never hold the response

  const ua = req.get("user-agent") ?? "";
  if (!MAYBE_BOT.test(ua) && LOOKS_LIKE_BROWSER.test(ua)) return;

  // Fire and forget — never make a visitor wait on analytics.
  fetch("https://truestat.io/api/crawl", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      site_key: "ts_your_site_key",
      user_agent: ua,
      path: req.path,
      ip: req.ip,
    }),
  }).catch(() => {});
});

Cloudflare Workers · Hono · Deno · Bun

Anything that hands your handler a plain Request.

npm install @truestat/ai-crawl

import { trackRequest } from "@truestat/ai-crawl/request";

export default {
  async fetch(request, env, ctx) {
    trackRequest("YOUR_SITE_KEY", request, ctx);
    return new Response("…");
  },
};

ctx is whatever your runtime calls its background-work handle — Cloudflare's ExecutionContext, or a bare waitUntil function elsewhere. Pass it and the report outlives the response, which matters here: a crawler's request finishes the moment you return, and without it the report can be cancelled mid-flight.

Once you have the response, trackResponse reports the status code too — a crawled page that 404s is how you learn an AI is following links that no longer exist:

const response = await handle(request);
trackResponse("YOUR_SITE_KEY", request, response, ctx);
return response;

Or write it yourself

The same thing without a dependency.

// Inside your fetch handler — Cloudflare Workers, Hono, Deno, Bun.

// Should this request be reported at all? A crawler either says so, or
// does not look like a browser. Both checks are loose on purpose —
// TrueStat classifies properly on its side, and a wasted request costs
// far less than a crawler that goes unrecorded.
//
// This is NOT about which browser a visitor uses; that is worked out
// from the user agent on TrueStat's side and shown in your dashboard.
// Here the only question is whether to send the request.
//
// Two things follow from keeping it this loose, and both are the point:
//
//   You never have to update this. It does not know WHICH crawler it is
//   looking at, so a crawler that did not exist when you pasted this is
//   still reported — TrueStat recognises it on arrival.
//
//   Your server does not post to us on every human pageview. Without
//   these two lines that is 20-30x the outbound requests, to send rows
//   the endpoint discards anyway.
const MAYBE_BOT = /bot|crawl|spider|\+http|compatible;|headless|preview/i;
const LOOKS_LIKE_BROWSER =
  /Mozilla\/5\.0 .*(Chrome|CriOS|Firefox|FxiOS|Safari|Edg|OPR|SamsungBrowser)\//;

export default {
  async fetch(request, env, ctx) {
    const ua = request.headers.get("user-agent") ?? "";

    if (MAYBE_BOT.test(ua) || !LOOKS_LIKE_BROWSER.test(ua)) {
      // `waitUntil` keeps the report alive past the response.
      ctx.waitUntil(fetch("https://truestat.io/api/crawl", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          site_key: "YOUR_SITE_KEY",
          user_agent: ua,
          path: new URL(request.url).pathname,
          ip: request.headers.get("cf-connecting-ip") ?? "",
        }),
      }).catch(() => {}));
    }

    return new Response("…");
  },
};

WordPress

Your child theme's functions.php, or a small site-specific plugin.

// Your child theme's functions.php, or a small site-specific plugin.
// Editing the parent theme means the next update deletes this.

add_action('init', function () {
    // Crawlers either say so, or do not look like a browser. Both are
    // checked loosely on purpose — TrueStat classifies properly on its side.
    $maybe_bot = '/bot|crawl|spider|\+http|compatible;|headless|preview/i';
    $browser = '/Mozilla\/5\.0 .*(Chrome|CriOS|Firefox|FxiOS|Safari|Edg|OPR|SamsungBrowser)\//';

    if (is_admin() || wp_doing_ajax() || wp_doing_cron()) return;

    $ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
    if (!preg_match($maybe_bot, $ua) && preg_match($browser, $ua)) return;

    $ip = explode(',', $_SERVER['HTTP_X_FORWARDED_FOR'] ?? '')[0]
        ?: ($_SERVER['REMOTE_ADDR'] ?? null);

    // Non-blocking — the visitor's page never waits on analytics.
    wp_remote_post('https://truestat.io/api/crawl', [
        'blocking' => false,
        'timeout'  => 2,
        'headers'  => ['Content-Type' => 'application/json'],
        'body'     => wp_json_encode([
            'site_key'   => 'ts_your_site_key',
            'user_agent' => $ua,
            'path'       => parse_url($_SERVER['REQUEST_URI'] ?? '/', PHP_URL_PATH),
            'ip'         => trim($ip),
        ]),
    ]);
});

Django

A middleware class, added to MIDDLEWARE in settings.

# Put this in its own module, then add it to MIDDLEWARE in settings.py.

import re, json, threading, urllib.request

# Crawlers either say so, or do not look like a browser. Both are checked
# loosely on purpose — TrueStat classifies properly on its side, and a
# wasted request costs far less than a crawler that goes unrecorded.
MAYBE_BOT = re.compile(r"bot|crawl|spider|\+http|compatible;|headless|preview", re.I)
LOOKS_LIKE_BROWSER = re.compile(r"Mozilla/5\.0 .*(Chrome|CriOS|Firefox|FxiOS|Safari|Edg|OPR|SamsungBrowser)/")


class TrueStatCrawlMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        response = self.get_response(request)
        ua = request.META.get("HTTP_USER_AGENT", "")
        if MAYBE_BOT.search(ua) or not LOOKS_LIKE_BROWSER.search(ua):
            self._report(ua, request)
        return response

    def _report(self, ua, request):
        fwd = request.META.get("HTTP_X_FORWARDED_FOR", "")
        ip = fwd.split(",")[0].strip() or request.META.get("REMOTE_ADDR")
        body = json.dumps({
            "site_key": "ts_your_site_key",
            "user_agent": ua,
            "path": request.path,
            "ip": ip,
        }).encode()
        req = urllib.request.Request(
            "https://truestat.io/api/crawl", data=body,
            headers={"Content-Type": "application/json"}, method="POST",
        )
        # In a thread — a visitor's page must never wait on analytics.
        threading.Thread(target=lambda: _swallow(req), daemon=True).start()


def _swallow(req):
    try:
        urllib.request.urlopen(req, timeout=2)
    except Exception:
        pass  # analytics must never break the page

Laravel

A middleware class, registered in your HTTP kernel.

<?php
// app/Http/Middleware/TrueStatCrawl.php
// Register it in bootstrap/app.php (Laravel 11+) or Kernel.php.

namespace App\Http\Middleware;

use Closure;
use Illuminate\Support\Facades\Http;

class TrueStatCrawl
{
    // Crawlers either say so, or do not look like a browser. Both are
    // checked loosely on purpose — TrueStat classifies properly on its
    // side, and a wasted request costs far less than a missed crawler.
    private const MAYBE_BOT = '/bot|crawl|spider|\+http|compatible;|headless|preview/i';
    private const LOOKS_LIKE_BROWSER = '/Mozilla\/5\.0 .*(Chrome|CriOS|Firefox|FxiOS|Safari|Edg|OPR|SamsungBrowser)\//';

    public function handle($request, Closure $next)
    {
        return $next($request);
    }

    /**
     * Runs AFTER the response has been sent to the visitor.
     *
     * PHP has no fire-and-forget HTTP call, so reporting inside handle()
     * would add its latency to the page. terminate() is Laravel's hook for
     * exactly this — the visitor already has their page by the time it runs.
     */
    public function terminate($request, $response)
    {
        $ua = $request->userAgent() ?? '';
        if (!preg_match(self::MAYBE_BOT, $ua) && preg_match(self::LOOKS_LIKE_BROWSER, $ua)) {
            return;
        }

        try {
            Http::timeout(2)->post('https://truestat.io/api/crawl', [
                'site_key' => 'ts_your_site_key',
                'user_agent' => $ua,
                'path' => $request->getPathInfo(),
                'status' => $response->getStatusCode(),
                'ip' => $request->ip(),
            ]);
        } catch (\Throwable $e) {
            // analytics must never break the page
        }
    }
}

Anything else

Plain HTTP. Any language, no package required — Go, Rails, Rust, a CDN worker, a reverse proxy.

# Plain HTTP — any language, no package required.
# Call this from wherever your server handles a request.

# `ip` is optional. Without it the visit is still recorded, just not
# verified as genuinely from that operator.

curl -X POST \
  https://truestat.io/api/crawl \
  -H "Content-Type: application/json" \
  -d '{
    "site_key": "ts_your_site_key",
    "user_agent": "<the crawler user-agent>",
    "path": "/the-page-requested",
    "ip": "<the crawler IP>"
  }'

# Send it in the background: a visitor's page must never wait on
# analytics, and a failure here must never reach them.
#
# Which requests to send. Two loose checks, not a list of crawler
# names — a name list goes stale every time an operator launches
# something new, and this code lives on your server, not ours.
#
#   MAYBE_BOT          = /bot|crawl|spider|\+http|compatible;|headless|preview/i
#   LOOKS_LIKE_BROWSER = /Mozilla\/5\.0 .*(Chrome|CriOS|Firefox|FxiOS|Safari|Edg|OPR|SamsungBrowser)\//
#
#   send if:  MAYBE_BOT.test(ua) || !LOOKS_LIKE_BROWSER.test(ua)
#
# In words: it says it is a bot, or it does not look like a browser.
# A real visitor on Chrome matches the second and is skipped, so you
# are not paying for a request on every human pageview.
#
# Also skip paths containing a dot, or every image costs a request.

Checking it works

curl -X POST https://truestat.io/api/crawl \
  -H "Content-Type: application/json" \
  -d '{
    "site_key": "ts_your_site_key",
    "user_agent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
    "path": "/install-test",
    "ip": "66.249.66.1"
  }'

A 202, with:

{"ok":true,"recorded":1,"skipped":0,"verified":1,"spoofed":0,"paused":false}

That IP is a real Googlebot address, so verified: 1 confirms the check is working end to end.

Then open the AI visibility panel. The visit appears within a minute.

Batching

Every crawler visit costs one request. On a heavily crawled site, collect them and send periodically instead:

const queue: Hit[] = [];

function track(hit: Hit) {
  queue.push(hit);
  if (queue.length >= 50) flush();
}

function flush() {
  if (!queue.length) return;
  const batch = queue.splice(0, 50);
  void fetch("https://truestat.io/api/crawl", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ site_key: SITE_KEY, hits: batch }),
  }).catch(() => {});
}

setInterval(flush, 10_000);

Set timestamp on each hit when you batch, or they all land at flush time and the timing of the crawl is lost.

Troubleshooting

recorded: 0, skipped: N — those requests were human. Normal unless you expected a crawler in that window.

404 unknown site key — the key does not match a site. Copy it from the install page.

400 invalid body — a required field is missing, or hits is longer than 50. The response names the field.

Nothing in the panel — check the response body rather than the status. A 202 with recorded: 0 means the call worked and nothing qualified.

Verified is always 0 — you are not sending ip, or the crawler's operator publishes no reverse records. See the table above.

Now go and read it

Crawler visits land in the AI & LLMs card on your dashboard, usually within seconds of the first one.

• What this panel shows — the four categories, and what each counts

• AI traffic by page — which of your pages they took

• How reliable is it — what "verified" means, and the honest limits