Recording crawlers from your server

The tracking script runs in a browser. Most AI crawlers do not use one — they request your HTML and never execute JavaScript, so the script never loads and the visit is never recorded.

On a real site the difference is large. Before server-side tracking was added, one site in our own account had recorded 4 crawler visits against 203 human ones over three days: two Bingbot hits and two test requests. No GPTBot, no ClaudeBot, no Googlebot, on a site those crawlers visit constantly.

If the AI panel looks emptier than you expect, this is why.

What you send

One HTTP request from your server, per crawler visit or in batches:

POST https://truestat.io/api/crawl
Content-Type: application/json

{
  "site_key": "ts_your_site_key",
  "hits": [
    {
      "user_agent": "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)",
      "path": "/pricing",
      "ip": "20.171.207.14",
      "status": 200
    }
  ]
}

Only site_key, user_agent and path are required. For a single visit you can skip the hits array and put the fields at the top level.

Up to 50 hits per request.

The response tells you if it worked

{ "ok": true, "recorded": 3, "skipped": 12, "verified": 1, "spoofed": 0 }

• recorded — crawler visits stored

• skipped — requests that were not crawlers, ignored rather than rejected

• verified — confirmed as genuinely from that operator

• spoofed — the user agent claimed an operator the IP does not belong to

skipped is the useful one during setup. If you send every request and see recorded: 0, skipped: 40, no crawler reached you in that window. If you see recorded: 0, skipped: 0, nothing arrived at all and the call is not firing.

Human traffic is ignored, not rejected

Send every request if that is simpler. Anything that is not a crawler is skipped rather than stored, so your human visits are never counted twice — the script already records those.

Verification

Passing ip lets us check the crawler is who it says it is. A user agent is a claim: anyone can run curl -A "GPTBot", and without checking, a scraper inflates your AI report.

The check is forward-confirmed reverse DNS, the method Google and Bing document. The IP's reverse record must belong to the operator, and that name must resolve forward to the same IP. The second step matters — a reverse record alone can be set by whoever controls the address.

Three outcomes:

Unverified is not suspicion. Several operators publish no reverse records at all, so their visits can never be more than unverified:

Sending ip is optional. Without it every visit is recorded as unverified.

Installing it

Express

const CRAWLERS =
  /bot|crawler|spider|slurp|gptbot|claudebot|anthropic|perplexity|applebot|bytespider|ccbot|facebookexternalhit|whatsapp|telegram|discord|slack/i;

app.use((req, res, next) => {
  next(); // never hold the response

  const ua = req.get("user-agent") ?? "";
  if (process.env.NODE_ENV !== "production") return;
  if (!CRAWLERS.test(ua)) return;
  if (req.path.includes(".") || req.path.startsWith("/api/")) return;

  fetch("https://truestat.io/api/crawl", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      site_key: process.env.TRUESTAT_SITE_KEY,
      user_agent: ua,
      path: req.path,
      ip: req.ip,
    }),
  }).catch(() => {});
});

Anything else

Plain HTTP, so any language works:

import requests, threading

def track(user_agent, path, ip):
    threading.Thread(target=lambda: requests.post(
        "https://truestat.io/api/crawl",
        json={"site_key": SITE_KEY, "user_agent": user_agent,
              "path": path, "ip": ip},
        timeout=2,
    ), daemon=True).start()

Send it in the background. A visitor's page should never wait on this.

Checking it works

curl -X POST https://truestat.io/api/crawl \
  -H "Content-Type: application/json" \
  -d '{
    "site_key": "ts_your_site_key",
    "user_agent": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)",
    "path": "/test",
    "ip": "66.249.66.1"
  }'

Expect {"ok":true,"recorded":1,"skipped":0,"verified":1,"spoofed":0}. That IP is a real Googlebot address, so verified: 1 confirms the DNS check is working end to end.

Then open the AI visibility panel. The visit appears within a minute.

Batching

Every crawler visit costs one request. On a heavily crawled site, collect them and send periodically instead:

const kuyruk: Hit[] = [];

function track(hit: Hit) {
  kuyruk.push(hit);
  if (kuyruk.length >= 50) bosalt();
}

function bosalt() {
  if (!kuyruk.length) return;
  const parti = kuyruk.splice(0, 50);
  void fetch("https://truestat.io/api/crawl", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ site_key: SITE_KEY, hits: parti }),
  }).catch(() => {});
}

setInterval(bosalt, 10_000);

Set timestamp on each hit when you batch, or they all land at flush time and the timing of the crawl is lost.

Troubleshooting

recorded: 0, skipped: N — those requests were human. Normal unless you expected a crawler in that window.

404 unknown site key — the key does not match a site. Copy it from the install page.

400 invalid body — a required field is missing, or hits is longer than 50. The response names the field.

Nothing in the panel — check the response body rather than the status. A 202 with recorded: 0 means the call worked and nothing qualified.

Verified is always 0 — you are not sending ip, or the crawler's operator publishes no reverse records. See the table above.