@devtune/ai-traffic

Measure which AI crawlers visit your site with a dependency-free sensor that filters at the edge and forwards only matched machine traffic to DevTune.

Install

pnpm add @devtune/ai-traffic

Create a project-scoped server-side ingest key and expose it only to your server runtime:

DEVTUNE_AI_TRAFFIC_INGEST_KEY=dt_ingest_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

On Vercel you can use a project ID instead of a key. See Vercel without an ingest key.

Quickstarts

Next.js

For Next.js 16, add proxy.ts. For older Next.js projects, use the same body in middleware.ts and export middleware() instead.

// proxy.ts
import { createDevTuneAiTrafficMiddleware } from "@devtune/ai-traffic";
import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";

const trackAiTraffic = createDevTuneAiTrafficMiddleware({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
});

export function proxy(request: NextRequest, event: NextFetchEvent) {
  trackAiTraffic(request, event);

  return NextResponse.next();
}

Express

import express from "express";
import { createDevTuneAiTrafficExpressMiddleware } from "@devtune/ai-traffic/express";

const app = express();

app.use(
  createDevTuneAiTrafficExpressMiddleware({
    ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  }),
);

The middleware calls next() immediately and records the actual response status after Express emits finish.

Node HTTP

import { createServer } from "node:http";

import { createDevTuneAiTrafficNodeHandler } from "@devtune/ai-traffic/node";

const trackAiTraffic = createDevTuneAiTrafficNodeHandler({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
});

createServer((request, response) => {
  trackAiTraffic(request, response);

  response.statusCode = 200;
  response.end("ok");
}).listen(3000);

The hook records the final response.statusCode without blocking or changing the response.

Vercel without an ingest key

On Vercel, the sensor can authenticate with each deployment's Vercel OIDC token instead of an ingest key, so there is no long-lived secret to store. Vercel linking must be enabled for your DevTune account while it is in beta.

1. In DevTune, open Settings › Integrations › Edge middleware and select Add Vercel project. Enter the Vercel team ID, shown as Team ID in the team's Settings › General, and the Vercel project ID, shown as Project ID in the project's Settings › General.

2. Copy the DevTune project ID from the same page and add it to the Vercel project's environment variables. It is not a secret.

   DEVTUNE_AI_TRAFFIC_PROJECT_ID=3f0c9a52-6b1e-4d7a-9c2f-1a2b3c4d5e6f
   

3. Pass it as projectId. Keep ingestKey while you migrate:

   const trackAiTraffic = createDevTuneAiTrafficMiddleware({
     projectId: process.env.DEVTUNE_AI_TRAFFIC_PROJECT_ID!,
     ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
   });
   

4. Deploy. Once the linked project shows Last seen in DevTune, remove ingestKey and delete the DEVTUNE_AI_TRAFFIC_INGEST_KEY variable.

The same projectId option works for the Express and Node HTTP adapters running on Vercel.

If you write your own adapter with projectId set, and it skips some requests or calls trackRequest() after the response, call the client's captureVercelOidcToken(request) on every request as it arrives, before any filtering. Vercel puts a current token on every request, so browser requests keep the token for pending sends current.

When both options are set, the sensor sends with the Vercel token and falls back to the ingest key if Vercel or DevTune cannot verify it. With only projectId, nothing is sent from a host that is not Vercel, and the sensor logs one warning per process. If a link stays Not seen yet after a deploy that received crawler traffic, check both Vercel IDs.

The token is read from each incoming request, never from the environment, and is sent only to Vercel to obtain a token for DevTune. That token is cached in memory until shortly before it expires and is sent only to the ingest endpoint.

Privacy by Design

The sensor filters requests where your application runs. It uses a cheap user-agent hint before consulting the AI bot registry, and only matched machine hits are eligible to be forwarded. It uses no cookies, no fingerprinting, and no full request logs.

Requests that a page makes for itself are never forwarded. A crawler that renders pages runs their JavaScript, and its browser then fires prefetches, client navigations and other fetch() calls from inside each page. Counting them would report crawls of pages it only fetched in the background; on DevTune's own site, one rendering crawler made three such requests to / for every page it loaded. The sensor recognises them from the fetch metadata the browser attaches, so no configuration is needed:

• Sec-Fetch-Dest is present and is not document, iframe, frame or fencedframe. empty marks a fetch() or XHR from a page, and values such as script or image mark subresources.
• Sec-Purpose or Purpose includes prefetch. This catches speculation-rules prefetches, which browsers send as document fetches, and marked prefetches from clients that send no Sec-Fetch-* headers.
• Next-Router-Prefetch is present. Next.js removes this header before middleware runs, so it only applies where the request reaches the sensor intact, such as the Express and Node adapters or a route handler. The fetch metadata above works in middleware too.

Crawlers that do not run a browser usually send no fetch metadata, and their requests are still recorded. The tradeoff: once an AI browser has loaded a page, its client navigations within your site are page-initiated fetches and are not counted. Its first page load is.

Browsers send fetch metadata only to HTTPS origins. On a site served over plain HTTP, page-initiated requests carry no Sec-Fetch-* headers, so the sensor cannot tell them apart from page loads and records them.

The exception is framework data endpoints, which the sensor skips by URL shape unless the browser marks the request as a page load (Sec-Fetch-Dest: document, iframe, frame or fencedframe). Several frameworks load a route's data from a separate URL during client navigation and link prefetching, and their servers never answer those URLs with a page:

• Next.js Pages Router: a path under /_next/data/ ending in .json, or any request carrying x-nextjs-data. Next.js middleware sees these requests at the page's own path, so there the header is what identifies them.
• Remix: any request with a _data query parameter. Without this rule, /pricing?_data=routes/pricing would be recorded as a crawl of /pricing once the query is removed.
• Remix and React Router single fetch: a path ending in .data, such as /pricing.data or /_root.data.
• React Router lazy route discovery: a path ending in /__manifest with a paths query parameter (React Router 7.10+) or p (earlier), such as /__manifest?paths=/pricing. A /__manifest page without it is still recorded.
• SvelteKit: a path ending in /__data.json or .html__data.json.
• Nuxt: a path ending in /_payload.json, or /_payload.js when JSON payloads are turned off.

The match is exact, so pages such as /data or /blog/__data-tips are still recorded. Other framework requests made from inside a page, such as Next.js App Router fetches, still depend on fetch metadata.

For a matched crawler, DevTune receives only:

• path
• origin plus path, with query parameters removed
• user agent
• response status when the adapter can know it
• timestamp
• the address the request arrived from, used to verify the crawler and then discarded
• for a signed agent, its Web Bot Auth signature material, used to verify the agent and then discarded (see Signed Agents)

Request bodies, cookies, unrelated headers, and query strings are never sent. Signature material is only forwarded for a request without a query, so a reset link, OAuth callback or search URL a signed agent follows never leaves the site. Only these request headers are ever forwarded with a signature: accept, accept-encoding, accept-language, cache-control, content-length, content-type, date, from, host and user-agent, exported as WEB_BOT_AUTH_FORWARDED_HEADERS. A request whose signature covers any other header, such as cookie, authorization or an API key, forwards no signature material at all, so credentials never leave the site. An address is sent only for a request whose user agent matches the AI bot registry, so ordinary browser traffic never reaches DevTune at all. Note the limit of that statement: the match is on the user agent, and anyone can send a crawler's user agent, so a person whose client does that would have their address sent. That is the same fact the verification exists to handle, and it is why the address is checked and discarded rather than stored. DevTune checks it against the claimed operator's published networks on receipt and keeps the verdict, not the address. Set trustedProxy: "none" to send no address at all; DevTune then records that traffic as unverified.

Bot Verification

A user agent is a claim, not an identity. Anything on the internet can send GPTBot/1.3, so a sensor that reports user agents alone reports what strangers say about themselves. The address the request arrived from is what makes the claim checkable.

The package only reads that address from a source your deployment declares as trustworthy, because a forgeable one verifies nothing:

x-forwarded-for is only read when you select "cloudfront" or pass a numeric offset, because any client can append to it. When it is read, entries are counted from the right, so a padded chain cannot push the trusted hops out of range. Values that do not parse as an IPv4 or IPv6 literal are dropped.

No address is sent when the declared proxy's or hop count's header is absent, or when an edge runtime has neither a socket nor a declared proxy. DevTune records that traffic as unverified rather than guessing.

const trackAiTraffic = createDevTuneAiTrafficMiddleware({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  trustedProxy: "cloudflare",
});

Signed Agents

Some agents sign each request with Web Bot Auth, an HTTP Message Signature (RFC 9421) that their operator's published key verifies. ChatGPT agent, Google's agent and DuckAssistBot do. A valid signature identifies the agent whatever network it runs on, so DevTune checks it alongside the address.

When a matched request carries all three of Signature, Signature-Input and Signature-Agent, its event includes a webBotAuth object with what DevTune needs to check the signature:

• the three headers, verbatim
• the request method
• the absolute request URL the agent sent. The Express and Node adapters take it from the request line and the incoming host without parsing it. Fetch-compatible runtimes such as Next.js have already normalised request.url, so there the sensor puts back the raw Host or X-Forwarded-Host the runtime built it from: a request signed for https://DOCS.example.com:443/guide forwards that, not https://docs.example.com/guide. The raw path is gone there, since the runtime may have rewritten /a/../guide as /guide, and behind a TLS-terminating proxy request.url can say http: for a request the agent sent over https:. So a request whose signature covers the scheme, path or query (@scheme, @path, @target-uri, @request-target or @query) forwards no signature on a Fetch runtime. Signatures over @authority and Signature-Agent, which is what Web Bot Auth agents sign today, still forward. The Express and Node adapters keep the raw path and forward either kind. A request without a Host header forwards no signature. A configured origin changes the event's url, never the signed URL.
• any other header the signature covers, by lowercase name, with repeated values combined as a, b. Only the headers in WEB_BOT_AUTH_FORWARDED_HEADERS are ever forwarded.

No configuration is needed, and the sensor never verifies, logs or stores any of it. DevTune verifies it on receipt and keeps the verdict, not the material. If any part exceeds DevTune's size limits, holds characters a request header cannot carry, or the signature covers a header outside WEB_BOT_AUTH_FORWARDED_HEADERS, the sensor leaves webBotAuth out rather than truncating it, because a truncated signature can never verify. That event, like any request without all three headers, is sent exactly as an unsigned one.

A signature is valid for a short window, and DevTune checks it against the event's time. The Express and Node adapters and withDevTuneAiTrafficRoute() record a request after its response, so they stamp a signed event with the time the request arrived. Unsigned events keep the time they are recorded. If you call trackRequest() yourself after the response, take client.signedArrivalTime(request) when the request arrives and pass it as the fourth argument:

const receivedAt = aiTraffic.signedArrivalTime(request);
const response = await render(request);

aiTraffic.trackRequest(request, event, response.status, receivedAt);

A request with a query string, even a bare trailing ?, forwards no signature material, and its event is sent exactly as an unsigned one. Query strings can carry reset tokens, OAuth codes and search terms, and the event's url leaves them out, so DevTune could never match a signature over one anyway. Such a request keeps its address verdict.

Detect Without Sending

The standalone classifiers need no ingest key and never make a network request or push data to DevTune:

import { detectAiCrawler, detectAiReferrer } from "@devtune/ai-traffic";

const crawler = detectAiCrawler(request);
const referrer = detectAiReferrer(request);

Each function also accepts the relevant string directly: a user-agent for detectAiCrawler() or a referrer URL for detectAiReferrer().

detectAiCrawler() returns the matched bot registry entry or null:

type AiCrawlerDetection = {
  uaPattern: string;
  llmPlatform: string;
  botClass: "training_crawler" | "index_bot" | "answer_fetcher" | "acting_agent";
  label: string;
  status?: "active" | "retired";
  asnHints?: number[] | null;
  notes?: string | null;
};

detectAiReferrer() returns the matched referrer registry entry or null:

type AiReferrerDetection = {
  hostname: string;
  llmPlatform: string;
  label: string;
};

Both use bundled registry snapshots by default, so they work offline and in CI. Pass registryEntries as the second argument when you need to classify against a pinned or private registry:

const match = detectAiCrawler(userAgent, {
  registryEntries: myRegistryEntries,
});

Version Reporting

Ingest and registry requests include an x-devtune-sdk-version header with the package version, so DevTune can show an upgrade notice when a newer release is available. It carries no visitor data.

Machine Traffic and Human Referrals

The sensor deliberately measures machine traffic only. DevTune gets human visits from AI products through its GA4 integration, where sessions, engagement, and conversions provide a richer picture than middleware referrer matching. The referrer detector is available for local classification, but the sensor adapters never forward human referral visits.

Production Notes

Batching and Runtime Lifetime

The default client configuration sends up to 10 matched events per unchanged ingest payload, with a 250 ms flush window for low-volume traffic. Set batchSize: 1 and flushIntervalMs: 0 to opt out and send each matched event immediately.

Express and Node servers are long-lived enough to benefit directly from the default. Next.js proxy and middleware pass the delayed send to event.waitUntil() when available, which prevents the response from waiting but may keep the middleware invocation alive for the short flush window. Use the explicit opt-out in short-lived runtimes that cannot reliably preserve delayed work, or when minimizing middleware duration matters more than request coalescing:

const trackAiTraffic = createDevTuneAiTrafficMiddleware({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  batchSize: 1,
  flushIntervalMs: 0,
});

Registry refreshes use waitUntil when provided, with a five-second fetch timeout. They run in the background and do not delay the response. Set waitUntilRegistryRefresh: false to opt out of keeping refresh work alive; short-lived runtimes may then stop it before it completes.

Rate-Limit Retries

A 429 from the ingest endpoint is the one failure the server tells us is temporary, so the batch is redelivered rather than dropped. The client waits for the response's Retry-After — delay-seconds or an HTTP date — and falls back to exponential backoff from 250 ms when the header is absent or unusable. The batch being retried is held intact, so events that arrive during a backoff are sent separately rather than folded into it.

Retries are bounded so a short-lived runtime cannot be held open by a throttled endpoint. The budget belongs to one flush, not to each batch: maxRetryAttempts defaults to 3 and maxRetryDelayMs caps each wait at 5000 ms, including a Retry-After longer than the cap, so a flush adds at most 15 s no matter how deep the queue is. When the budget is spent the current batch is dropped with the usual sampled warning and the drain stops; whatever is behind it stays queued for the next flush, which gets a fresh budget. maxRetryAttempts: 0 disables waiting entirely: a 429 drops the batch on its first rejection, as it did before this behaviour existed. The drain still stops there rather than dropping every batch behind it, so the rest stays queued. Tune both:

const trackAiTraffic = createDevTuneAiTrafficMiddleware({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  maxRetryAttempts: 2,
  maxRetryDelayMs: 2_000,
});

Because backing off lets the queue drain slower than traffic arrives, the queue itself is bounded. maxQueuedEvents defaults to 1000; past it the oldest events are shed with a sampled warning, so a sustained throttle cannot grow memory without limit.

Non-429 responses and network errors are unchanged: the batch is dropped after a single attempt, so a hard outage never keeps the queue alive. The one exception is a 503 to a batch sent with a Vercel token and no ingest key configured: DevTune could not reach Vercel to verify the token, so it is retried like a 429 within the same budget.

Status Semantics

Express and Node adapters observe the completed response and report its actual status. Next.js proxy and middleware cannot observe the final route status after pass-through, so the convenience middleware omits status instead of guessing a 200.

When a Next.js proxy returns a response directly, use the client and pass the known status:

import { createDevTuneAiTraffic } from "@devtune/ai-traffic";
import type { NextFetchEvent, NextRequest } from "next/server";

const aiTraffic = createDevTuneAiTraffic({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  defaultStatus: null,
});

export function proxy(request: NextRequest, event: NextFetchEvent) {
  const response = new Response("Forbidden", { status: 403 });

  aiTraffic.trackRequest(request, event, response.status);

  return response;
}

Use withDevTuneAiTrafficRoute() where you own a Fetch-compatible route handler and want its exact response status captured automatically. notFoundPathPatterns and statusResolver remain available for applications that can provide additional status knowledge.

Registry Refresh and Fallback

Version 0.5.0 adds Amazon and DuckDuckGo registry support, along with explicit Claude Code, Gemini CLI and OpenCode retry detection. Recognition is a self-reported client claim. Client-run tools can use the user's network, so an address outside a hosted crawler list does not by itself prove an identity mismatch. DevTune performs identity checks on receipt. Generic browser or curl traffic cannot reliably identify an AI client or its model provider.

From 0.5.0, new platform names and traffic classes can arrive through the remote registry without a package upgrade. DevTune handles classification and identity verification on its servers. The SDK keeps validating match patterns before using them.

Clients start with the bundled snapshot. The first request, then the first request after the one-hour cache expires, triggers a background refresh from https://devtune.ai/api/v1/llm-traffic/registry. This includes ordinary browser requests, but the registry GET contains no visitor data and only matching machine requests are sent to ingest. Node, Express and middleware use the same policy. Concurrent requests share one refresh, and ETag avoids downloading an unchanged registry.

A failed refresh keeps the last working registry, or the bundled snapshot before the first successful refresh. Retries wait for the next cache interval. Requests use the currently cached rules while a refresh is in progress, so a newly added bot may be missed until that refresh completes. Create the client or middleware once per process or runtime instance to reuse its cache.

Upgrades can still be needed for address extraction, hosting adapters, security fixes or event-format changes. Standalone detectAiCrawler() uses its bundled snapshot unless you supply registryEntries; it does not refresh automatically.

User-agent matching is a conservative signal. Some agents spoof ordinary browsers or require network-level signals, so reported counts are a floor rather than exact bot truth. Cloudflare-proxied sites should prefer DevTune's Cloudflare pull integration when available because it can combine request, bot-score, and network signals without running middleware.

Forwarded Origins

Express and Node adapters build event URLs from the request protocol and host, preferring the first value in standard X-Forwarded-Proto and X-Forwarded-Host chains. Only accept those headers from a trusted proxy. If they are unavailable or not trustworthy in your deployment, pass a fixed origin:

const trackAiTraffic = createDevTuneAiTrafficNodeHandler({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  origin: "https://www.example.com",
});

Failure Behavior

Registry refresh and ingest sends are fire-and-forget and guarded. Network failures may drop telemetry, but they do not block, throw into, or alter the customer response. The ingest endpoint and batched wire format are unchanged.

Documentation

• Install the edge middleware
• Choose the right AI Demand sensors

License

MIT © 2026 PLGeek Ltd.

CloudFront

When CloudFront connects directly to the application running the sensor, add trustedProxy: "cloudfront" to your existing middleware options. No CloudFront origin request policy change is needed.

CloudFront appends the viewer address to X-Forwarded-For, after any entries the viewer supplied. The sensor reads that rightmost entry and ignores earlier ones. See AWS's documented custom-origin behavior.

Only use this setting when requests reach the app through your trusted CloudFront distribution. If an ALB or another proxy between CloudFront and the app appends addresses, use a numeric trustedProxy offset instead: 0 selects the rightmost entry, 1 skips one appended address, and so on. Confirm the actual header chain before selecting an offset. Code executing inside CloudFront itself is a different integration; this setting is for middleware at the origin.