@crawtir/sdk

The TypeScript client for the Crawtir web crawling and extraction API. Scrape pages, crawl sites, discover URLs, extract structured fields, capture rendered UI, find public business leads, and search previously discovered companies.

Requires Node.js 22 or later. The package includes TypeScript declarations and works with any compatible Crawtir API deployment.

Install and connect

npm install @crawtir/sdk

Obtain an API key from the operator of your Crawtir service. Set CRAWTIR_API_URL to its HTTPS origin, without /v1.

import { Crawtir, CrawtirError } from '@crawtir/sdk';

const crawtir = new Crawtir({
  apiKey: process.env.CRAWTIR_API_KEY!,
  baseUrl: process.env.CRAWTIR_API_URL!, // e.g. https://api.example.com
  timeout: 120_000, // optional; ordinary request timeout in milliseconds
  retries: 2,       // optional; GET requests only
});

Call the SDK from a backend so your Crawtir key and any AI provider key stay private. Do not put them in a browser bundle or commit them to source control.

Methods

All methods authenticate with your Crawtir API key. searchLeads() additionally requires your own OpenAI or OpenRouter key on each call. A Crawtir API key does not include use of the service operator's AI account.

Scrape one page

const page = await crawtir.scrape({
  url: 'https://example.com/about',
  formats: ['markdown', 'html'],
  onlyMainContent: true,
  renderJavaScript: 'auto',
  timeout: 30_000,
});

console.log(page.title, page.statusCode);
console.log(page.markdown);
console.log(page.links);

scrape() returns a PageResult with url, canonicalUrl, title, description, metadata, links, statusCode, extractedAt, and durationMs. markdown and html are present when requested. A slow scrape can become an asynchronous job; the SDK polls that job and still resolves to a page.

The constructor's timeout controls the SDK's HTTP request and polling wait. The scrape input's timeout controls page processing.

Crawl multiple pages

crawl() starts an asynchronous job and returns its ID. Poll getCrawl() until its status is terminal, then page through getPages().

const job = await crawtir.crawl({
  url: 'https://example.com',
  limit: 50,
  maxDepth: 3,
  includePaths: ['/docs'],
  excludePaths: ['/docs/archive'],
  formats: ['markdown'],
});

let status: Record<string, unknown>;
for (;;) {
  status = await crawtir.getCrawl(job.id);
  if (['completed', 'failed', 'cancelled'].includes(String(status.status))) break;
  await new Promise((resolve) => setTimeout(resolve, 2_000));
}
if (status.status !== 'completed') {
  throw new Error(`Crawl ended with status ${String(status.status)}`);
}

for (let pageNumber = 1; ; pageNumber++) {
  const result = await crawtir.getPages(job.id, pageNumber, 100);
  for (const page of result.data) {
    console.log(page.url, page.title, page.markdown);
  }
  if (result.data.length < result.limit) break;
}

limit defaults to 100 pages (schema range 1–5,000; the deployment may enforce a lower cap). maxDepth defaults to 3 (range 0–20). includePaths and excludePaths filter paths and allow up to 100 entries each. Crawl input also accepts scrape options. getPages() defaults to page 1 with 20 results and permits up to 100 per request. Call await crawtir.cancelCrawl(job.id) to request cancellation.

Crawls follow links on the starting host and its www or non-www variant. They do not automatically include other subdomains such as blog.example.com. Starting URLs may redirect between the two host variants without ending the crawl.

Map a site or scrape known URLs

Use map() for URL discovery and batchScrape() for URLs you already know. Both return job IDs.

const mapJob = await crawtir.map({
  url: 'https://example.com',
  limit: 100,
  maxDepth: 2,
});

const batchJob = await crawtir.batchScrape({
  urls: [
    'https://example.com/about',
    'https://example.com/contact',
  ],
  formats: ['markdown'],
});

console.log(mapJob.id, batchJob.id);

map() defaults to 100 URLs and depth 2; the schema allows 1–5,000 URLs and depth 0–5, subject to the deployment's cap. batchScrape() accepts 1–500 URLs and the scrape options except url. The SDK does not currently have getMap() or getBatch() helpers. Read these jobs through the HTTP API with the same bearer key: GET /v1/map/:id, GET /v1/map/:id/pages, GET /v1/batch/:id, and GET /v1/batch/:id/pages.

Extract structured fields

Give extract() a field schema. Optional CSS selectors indicate where to read each field.

const data = await crawtir.extract({
  url: 'https://example.com/contact',
  schema: {
    heading: 'string',
    email: 'string|null',
    links: 'string[]',
  },
  selectors: {
    heading: 'h1',
    email: 'a[href^="mailto:"]',
    links: 'main a',
  },
});

console.log(data.heading, data.email, data.links);

Field types are string, string|null, and string[]. Supply 1–50 fields; names must start with a letter and contain only letters, digits, or underscores. The result is a record of extracted values. useAI?: boolean is available when the target deployment has extraction AI configured; it is separate from the caller-supplied provider used by searchLeads(). The SDK polls slow extraction jobs.

Capture rendered UI across a site

captureUi() visits multiple pages on the starting host and its www or non-www variant, then emits each page as it arrives. Before capturing a page, the browser scrolls through it to reveal lazy sections and images. A page contains rendered html, discovered links, visible controls, a JPEG viewport screenshot encoded as base64, viewport, and assets with the original absolute URL, content type, and base64 body. These assets include fetched images, CSS, fonts, and scripts; the screenshot is a separate preview, not a replacement for image files. Crawtir also scans HTML and CSS for resources the browser did not request in that viewport, including lazy image references and unused font variants, and fetches them within the asset limits. Crawtir streams these values directly and does not store the captured payload.

Set output: 'project' to receive a flat HTML filename for each page (index.html, about.html, blog--post.html), a separate combined stylesheet, and local paths for captured images, fonts, and scripts. page.project.html has local references; page.html remains the original rendered HTML. Each asset has url (the original request), resolvedUrl (the final URL after redirects, or the original URL), path, kind, content type, and base64 body. page.project.assetMap repeats the URL-to-path mapping without bytes for importers. Write the page, stylesheet, and assets at those paths to create a local project; no ZIP is produced. The stylesheet preserves CSS block order, while external scripts stay as separate files in their original tag order. The default output: 'raw' preserves the earlier response shape.

Use assetPathStyle: 'clean' for categorized paths under assets/images/, assets/js/, assets/css/, assets/fonts/, and assets/other/. The default source style keeps source host and directory structure, which can better support scripts that load relative chunks at runtime. excludeResourceKinds accepts challenge, analytics, and infrastructure. Classification is conservative: ordinary image, font, and script CDNs are site assets and remain included. Exclusions remove matching resources from the payload and project HTML; excluded resources are reported in skippedAssets.

import { mkdir, writeFile } from 'node:fs/promises';
import { dirname, join } from 'node:path';

const root = './captured-site';
async function save(path: string, bytes: string | Buffer) {
  const destination = join(root, path);
  await mkdir(dirname(destination), { recursive: true });
  await writeFile(destination, bytes);
}

for await (const event of crawtir.captureUi({
  url: 'https://example.com',
  output: 'project',
  assetPathStyle: 'clean',
  excludeResourceKinds: ['challenge', 'analytics'],
  limit: 20,
  maxDepth: 3,
})) {
  if (event.type === 'page' && event.page.project) {
    const { project, assets } = event.page;
    await save(project.path, project.html);
    if (project.stylesheet)
      await save(project.stylesheet.path, Buffer.from(project.stylesheet.base64, 'base64'));
    for (const asset of assets)
      if (asset.path) await save(asset.path, Buffer.from(asset.base64, 'base64'));
    for (const failure of event.page.skippedAssets ?? [])
      console.warn(failure.category, failure.url, failure.reason, failure.contentType, failure.referringPage);
    for (const script of project.scripts)
      console.log(script.originalUrl, script.path, script.captured, script.locallyReferenced, script.executionStatus);
    if (project.warnings.length) console.warn(project.warnings);
  }
}

Page links are rewritten to flat filenames. HTML references include ordinary and lazy image attributes, <picture> sources, srcset, scripts, icons, and preload links; CSS url() and @import references are rewritten to local asset paths. project.scripts lists document script tags in order, followed by captured network scripts not present as tags. It preserves async, defer, and type metadata and identifies Webflow, GSAP, ScrollTrigger, and jQuery signals. captured means Crawtir has the script bytes; locallyReferenced means the rewritten HTML points to the file. executionStatus: 'not_verified' means a local runtime check is still needed. A dynamically loaded chunk may be captured without a document tag. Clean paths can break a script that computes relative URLs or loads chunks dynamically; the project reports a warning for this case. API calls, authentication, forms, and site-specific runtime logic may also depend on the original service. Serve the saved files over HTTP to test behavior; browsers may restrict scripts and fonts opened directly with file://.

Strict completeness reports

Set output: 'project' and requireComplete: true to make the stream end with an incomplete event when an in-scope page fails, discovery hits the page limit, a required local file is absent, an asset remains remote without an explicit allowance, or a requested interaction check fails. Earlier page events remain available to inspect, but do not treat them as a successful import until the terminal event arrives. incomplete.report.gaps explains each failure. A successful run ends with complete and report.status === 'complete'. Without strict mode, complete.status can still be incomplete; inspect it.

for await (const event of crawtir.captureUi({
  url: 'https://example.com',
  output: 'project',
  requireComplete: true,
  domains: ['docs.example.net'], // additional exact page host; root and its www variant are included
  limit: 50,
  maxDepth: 3,
  externalAssetHosts: ['video.example.net'], // deliberately left remote
  interactionChecks: [{
    name: 'mobile-menu',
    pagePath: 'index.html',
    selector: '.w-nav-button',
    action: 'click',
    expectSelector: '.w-nav-button.w--open',
  }],
})) {
  if (event.type === 'page') {
    // Collect files, but commit the import only after the terminal result.
  }
  if (event.type === 'incomplete') {
    console.error(event.report.gaps);
    // Discard or keep the staged files for debugging.
  }
  if (event.type === 'complete') console.log(event.report?.status);
}

The report records the root and additional page hosts, path filters, page limit, depth, and captured states (initial_render, scroll_reveal, and named checks). It lists resource outcomes (captured_locally, intentionally_external, excluded, blocked, failed, not_verified) with URLs and reasons, plus local reference checks. Additional domains are exact hostnames; no wildcard or automatic sibling-domain crawl is implied. externalAssetHosts is an explicit allowance for remote assets, separate from page domains. After all pages arrive, the local preview runs each listed click or hover check against the collected project files, with outside requests blocked. Checks pass only when expectSelector becomes visible. Preview verification has a 128 MB memory budget; checks beyond it are reported not_verified. Unlisted interactions remain not_verified; this mode does not promise server-backed behavior or undiscovered routes. An importer should also run its own preview after writing the complete project.

Every skippedAssets entry has a URL, reason, content type (unknown if no response was received), referring page, and category: blocked, unsupported_format, size_limit, capture_failure, or excluded. Check these entries and missingAssetUrls before relying on an offline copy. For a difficult import, retain one page event with project.html, project.assetMap, project.scripts, skippedAssets, and a short list of interactions that fail in the local preview. Remove tokens and private URLs before sharing diagnostics.

for await (const event of crawtir.captureUi({
  url: 'https://example.com',
  limit: 20,
  maxDepth: 3,
  width: 1440,
  height: 900,
  excludePaths: ['/private'],
})) {
  switch (event.type) {
    case 'page':
      console.log(event.sequence, event.depth, event.page.url);
      console.log(event.page.controls.length, event.page.assets.length);
      for (const asset of event.page.assets) {
        const bytes = Buffer.from(asset.base64, 'base64'); // Node.js example
        console.log(asset.url, asset.contentType, bytes.length);
        // Use asset.url to match references in page.html or CSS.
      }
      if (event.page.missingAssetUrls.length)
        console.warn('Uncaptured references:', event.page.missingAssetUrls);
      // Persist or transform page.html, page.screenshot, and assets in your app.
      break;
    case 'page_error':
      console.warn(event.url, event.error);
      break;
    case 'complete':
      console.log(event.pages, event.errors, event.discovered);
      break;
  }
}

Events are started, page, page_error, heartbeat, complete, and in strict mode incomplete. A stream-ending error throws CrawtirError. limit defaults to 20 and permits up to 100 pages; maxDepth defaults to 3 and permits 0–10. Viewport width is 320–1,920 pixels (default 1,440), and height is 480–1,200 pixels (default 900). Asset bodies are capped at 8 MB each and 40 MB per page; the reference scan checks at most 320 additional URLs per page. page.assetReferences lists URLs referenced by rendered HTML and CSS, page.missingAssetUrls lists references absent from page.assets, and page.skippedAssets explains failed or oversized resources. resolvedUrl equals the final URL after redirects or the original URL when there was no redirect. This captures public UI and files, not server-side behavior, authenticated pages, or every state reachable through user interaction.

Find leads on the public web

searchLeads() takes a description of the business making the search and a target market. Crawtir plans searches, discovers public websites, checks business fit with your supplied AI provider, and streams qualified results with source links. Coverage depends on available search sources and crawlable pages. A lead is an observed match, not a guarantee of identity, current employment, or email deliverability.

for await (const event of crawtir.searchLeads({
  business: 'We supply event venues and fundraising payment tools to organizations',
  location: 'United Kingdom',
  limit: 5,
  provider: {
    name: 'openrouter',
    apiKey: process.env.OPENROUTER_API_KEY!,
    model: 'openai/gpt-4.1-mini',
  },
})) {
  switch (event.type) {
    case 'plan':
      console.log('Searches:', event.plan.searches);
      break;
    case 'search':
      console.log(event.query, event.found);
      break;
    case 'lead':
      console.log(event.lead.name, event.lead.website, event.lead.sourceUrl);
      break;
    case 'warning':
      console.warn(event.source, event.error);
      break;
    case 'complete':
      console.log('Leads:', event.leads, 'Pages used:', event.pages);
      break;
  }
}

For OpenAI, use provider.name: 'openai' with your OpenAI key and a supported model. Supply provider on every SDK or API lead search. The SDK and API require HTTPS when a provider key is sent. Crawtir encrypts that key for its short-lived processing queue and does not store it in PostgreSQL. Keep it in your own secret storage and avoid logging request bodies that contain it.

The stream emits started, plan, search, warning, lead, heartbeat, and complete. plan.searches shows the generated queries. complete.pages counts visited pages, including candidates that did not become leads. Consume the stream through complete; the SDK throws if it ends early. The server can run either streaming method for up to 15 minutes, so allow for a long-lived connection in your application and reverse proxy.

Return only the fields you need

fields controls each lead event. Choose from name, website, pageUrl, email, phone, summary, contactSourceUrl, searchQuery, fit, reason, and evidence. sourceUrl is always included for verification. Put a field in requiredFields to skip matches where that value is unavailable; each required field must also appear in fields.

for await (const event of crawtir.searchLeads({
  business: 'We provide booking and payment software for event venues',
  location: 'Canada',
  limit: 10,
  fields: ['email'],
  requiredFields: ['email'],
  contactPageLimit: 2,
  provider: {
    name: 'openai',
    apiKey: process.env.OPENAI_API_KEY!,
    model: 'gpt-4.1-mini',
  },
})) {
  if (event.type === 'lead') {
    // Shape: { sourceUrl: string, email: string | null }
    console.log(event.lead.email, event.lead.sourceUrl);
  }
}

An email is an address observed on a public page. Crawtir does not verify mailbox delivery. requiredFields: ['email'] excludes leads without a public email; it does not run an email-finder cascade. This workflow finds businesses and public contacts, not a dedicated people-by-role database.

Search the company index

searchCompanies() queries company records previously discovered by lead searches. It uses keyword search over the shared index and does not call an AI provider. The index grows as public business websites are qualified; it is not a complete web or people index.

const { data, count } = await crawtir.searchCompanies('event venue', 20);

for (const company of data) {
  console.log({
    name: company.name,
    website: company.website,
    source: company.sourceUrl,
    email: company.email,
    emailStatus: company.emailStatus,
    firstSeenAt: company.firstSeenAt,
    lastSeenAt: company.lastSeenAt,
  });
}
console.log('Returned:', count);

The query must be 2–200 characters. limit defaults to 20 and permits 1–50. An IndexedCompany can include id, domain, website, name, summary, email, emailSourceUrl, emailStatus, phone, sourceUrl, sourceTitle, firstSeenAt, and lastSeenAt. emailStatus: 'observed' means an email appeared on a crawled page; 'unavailable' means no public email was found. count is the number returned in this response, not the index total.

Errors, retries, and timeouts

HTTP errors and terminal streaming errors use CrawtirError, with message, numeric status, and optional details. Network and abort errors can also come from the runtime's fetch.

try {
  const page = await crawtir.scrape({ url: 'https://example.com' });
  console.log(page.title);
} catch (error) {
  if (error instanceof CrawtirError) {
    console.error('Crawtir request failed:', error.status, error.message);
    console.error('Details:', error.details);
  } else {
    throw error;
  }
}

Ordinary GET requests retry transient network failures and HTTP 429, 502, 503, and 504 by default, with two retries and increasing delay. Constructor retries changes this. Writes and streams are not retried automatically, avoiding duplicate jobs or searches. The default constructor timeout is 120 seconds for ordinary requests. Streaming requests use at least a 15-minute HTTP timeout; the server also enforces a 15-minute stream deadline. Align your application and proxy timeouts with the work requested.

TypeScript imports

The package exports Crawtir, CrawtirError, and types including ScrapeInput, CrawlInput, MapInput, BatchInput, ExtractInput, PageResult, UiCaptureInput, UiCapturePage, UiCaptureEvent, LeadSearchInput, LeadSearchPlan, LeadResult, LeadSearchEvent, LeadField, SelectedLeadResult, and IndexedCompany.

import { Crawtir, CrawtirError } from '@crawtir/sdk';
import type { PageResult, LeadSearchEvent, IndexedCompany } from '@crawtir/sdk';

For matching HTTP routes, authentication, and the OpenAPI reference, visit /docs on your Crawtir API deployment.