@datafast/ai-crawl

Server-side AI crawler tracking for DataFast.

Use this package to see when known AI assistants, AI search systems, or training crawlers request pages on your website. It is not a replacement for DataFast web analytics and does not track human visitors, sessions, conversions, attribution, or browser pageviews.

Full setup guide: DataFast bot traffic tracking docs

Crawler directory: See all crawlers tracked by DataFast

Installation

npm install @datafast/ai-crawl

How It Works

The package runs on your server, middleware, edge function, or worker. For each incoming request, it checks the user agent and path locally. If the request is not a likely AI/search crawler document request, nothing is sent to DataFast.

When a likely crawler requests a document page, the package sends a small event to DataFast's AI crawl endpoint. In runtimes that support background tasks, such as Vercel and Cloudflare, it uses waitUntil so your page response does not wait for DataFast.

Crawler-facing discovery files are tracked when requested by known crawlers. This includes robots.txt, llms.txt, llms-full.txt, and sitemap XML files.

The package is only a cost and safety pre-filter. DataFast's servers are the source of truth for provider classification, category, IP verification, confidence, and crawler list updates. This means DataFast can update crawler user agents and IP ranges without requiring you to upgrade this package for every crawler change.

Status code is optional. The most important signal is the page the AI tried to crawl. Status code is captured only when the backend exposes it without adding latency.

Important: when you pass a Vercel, Cloudflare, or Worker context to trackAICrawlerRequest, do not await it. The package schedules the tracking request with waitUntil and returns immediately.

Tutorials

Next.js / Vercel Proxy

Create or update proxy.ts (previously called middleware.ts / middleware.js in older Next.js versions):

// proxy.ts
import {
  NextResponse,
  type NextFetchEvent,
  type NextRequest,
} from "next/server";
import { trackAICrawlerRequest } from "@datafast/ai-crawl";

export function proxy(request: NextRequest, event: NextFetchEvent) {
  trackAICrawlerRequest(request, event, {
    websiteId: "dfid_your_tracking_id",
  });

  return NextResponse.next();
}

export const config = {
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};

This is non-blocking. The package schedules the DataFast request with event.waitUntil, then your app continues returning the page.

Next.js Proxy runs before the final page response, so status code is usually unknown and will be stored as 0.

Cloudflare Pages

Install the package, then create a Pages Functions middleware file:

// functions/_middleware.ts
import { trackAICrawlerRequest } from "@datafast/ai-crawl";

export async function onRequest(context) {
  trackAICrawlerRequest(context.request, context, {
    websiteId: "dfid_your_tracking_id",
  });

  return context.next();
}

Cloudflare Pages provides context.waitUntil, so the DataFast request runs in the background while context.next() continues to your page or static asset.

Do not await trackAICrawlerRequest(...) here. Call it, then immediately return context.next().

For better Cloudflare billing/performance hygiene, also exclude obvious static assets from Function invocation when your framework does not generate this automatically. Add _routes.json to your build output:

{
  "version": 1,
  "include": ["/*"],
  "exclude": [
    "/assets/*",
    "/static/*",
    "/_next/*",
    "/favicon.ico",
    "/*.css",
    "/*.js",
    "/*.png",
    "/*.jpg",
    "/*.svg",
    "/*.ico",
    "/*.woff2"
  ]
}

The package still filters static assets locally, but _routes.json prevents Cloudflare from invoking your Function for the most obvious asset requests. Do not exclude crawler-facing files such as /robots.txt, /llms.txt, /llms-full.txt, or /sitemap.xml if you want to see crawler activity on those files.

Cloudflare docs: Pages middleware, Pages Functions API, routing and _routes.json.

Cloudflare Workers

Wrap your Worker fetch handler:

import { withAICrawlerTracking } from "@datafast/ai-crawl";

export default {
  fetch: withAICrawlerTracking(
    async (request: Request, env: Env, ctx: ExecutionContext) => {
      return fetch(request);
    },
    {
      websiteId: "dfid_your_tracking_id",
    },
  ),
};

This captures status code because the wrapper sees the Response your handler already created. It still uses ctx.waitUntil when available, so the tracking request is best-effort and non-blocking.

Custom Docker, Cloud Run, or reverse proxies

Most frameworks expose the public request URL automatically. If your runtime instead builds request.url with an internal hostname such as localhost or 0.0.0.0, set publicOrigin once:

trackAICrawlerRequest(request, context, {
  websiteId: "dfid_your_tracking_id",
  publicOrigin: "https://example.com",
});

DataFast keeps the original path and query string, replaces only the internal origin, and still validates the resulting public hostname against your website configuration.

Express

import express from "express";
import { createExpressAICrawlerMiddleware } from "@datafast/ai-crawl";

const app = express();

app.use(
  createExpressAICrawlerMiddleware({
    websiteId: "dfid_your_tracking_id",
  }),
);

The Express middleware attaches a finish listener and sends status code after the response has already been sent to the crawler. It calls next() immediately and does not wait for DataFast before your app continues.

Hono

import { Hono } from "hono";
import { trackAICrawlerResponse } from "@datafast/ai-crawl";

const app = new Hono();

app.use("*", async (c, next) => {
  await next();

  trackAICrawlerResponse(c.req.raw, c.res, c.executionCtx, {
    websiteId: "dfid_your_tracking_id",
  });
});

This is useful on Cloudflare Workers or other Hono runtimes where you can access the final response and execution context.

Generic Request / Response Handler

If your backend gives you a standard Request and Response, use:

import { trackAICrawlerResponse } from "@datafast/ai-crawl";

export async function handleRequest(request, context) {
  const response = await yourAppHandler(request);

  trackAICrawlerResponse(request, response, context, {
    websiteId: "dfid_your_tracking_id",
  });

  return response;
}

If your backend only gives you the request before the app response exists, use request-only tracking:

import { trackAICrawlerRequest } from "@datafast/ai-crawl";

export function middleware(request, context) {
  trackAICrawlerRequest(request, context, {
    websiteId: "dfid_your_tracking_id",
  });

  return next();
}

Request-only tracking is enough to know which page the AI tried to crawl. Response-aware tracking only adds status code when it is easy to get.

Optional Request Authentication

Create a Bot traffic token in your website's Bot traffic settings, then add it to the same server-side configuration:

trackAICrawlerRequest(request, event, {
  websiteId: "dfid_your_tracking_id",
  authToken: process.env.DATAFAST_BOT_TOKEN,
});

Keep the token on your server and out of browser code. Token validation only affects request acceptance after you enable Reject unauthenticated requests in Bot traffic settings. Rotating a token invalidates the previous token immediately, and deleting it automatically turns request authentication off.

Filtering

By default, the package tracks every relevant crawler category it recognizes, including AI answer fetchers, search/indexing crawlers, training crawlers, and other AI crawler traffic.

You can opt out of specific categories:

trackAICrawlerRequest(request, event, {
  websiteId: "dfid_your_tracking_id",

  // Optional category opt-outs. All default to false.
  disableAnswerFetch: true,
  disableSearchCrawlers: true,
  disableTrainingCrawlers: true,
  disableOtherCrawlers: true,
});

For backward compatibility, includeSearchCrawlers: false also disables search/indexing crawlers. Prefer disableSearchCrawlers: true for new installs.

What Gets Tracked

The tracker only sends likely AI/search crawler document requests to DataFast. Normal human traffic, API routes, framework internals, and generic static assets such as CSS, JavaScript, images, fonts, maps, PDFs, and media files are ignored before any network request is made.

Crawler-facing files are intentionally not treated as generic static assets. DataFast tracks known crawler requests to files such as /robots.txt, /llms.txt, /llms-full.txt, and sitemap XML files, because those files are often requested specifically by search bots, AI assistants, and AI infrastructure.

It stores provider, agent, category, page, hostname, status code when available, source, and confidence.

OpenAI / ChatGPT-User  -> answer_fetch
OpenAI / OAI-SearchBot -> search_index
OpenAI / GPTBot        -> training
Googlebot              -> search_index
Google-NotebookLM      -> answer_fetch
Google-Extended        -> training
Google-CloudVertexBot  -> training
PerplexityBot          -> search_index
ClaudeBot              -> training
Bingbot                -> search_index
Applebot               -> training
Bytespider             -> training
CCBot                  -> training
xAI-SearchBot          -> answer_fetch
Grok-DeepSearch        -> answer_fetch
GrokBot / xAI-Bot      -> ai_crawler

License

MIT