@expandai/sdk

Official TypeScript SDK for the ExpandAI API.

Built on Effect, with both a class-based client for promise-style usage and an Effect.Service for full Effect integration.

Install

Requires Node.js 20.3 or newer.

npm install @expandai/sdk
# or
bun add @expandai/sdk
# or
pnpm add @expandai/sdk

Internal staging builds are available without changing the production latest tag:

npm install @expandai/sdk@staging

Pass the staging API URL through the client's baseUrl; selecting the npm tag does not change the default production endpoint.

Authentication

Set EXPAND_API_KEY in your environment, or pass apiKey explicitly when constructing the client.

export EXPAND_API_KEY=sk_...

Quick start

Promise API

import { ExpandClient } from '@expandai/sdk'

const expand = new ExpandClient()

try {
    const result = await expand.fetch({ url: 'https://example.com' })
    console.log(result)
} finally {
    await expand.close()
}

Reuse a client for the lifetime of your application, then call close() during shutdown. Closing aborts active requests, ends Crawl iterators and polling, and releases this client's connections without closing another client's pool. Repeated closes are safe; later requests on a closed client reject. Effect service consumers retain the existing scoped-layer cleanup.

Constructor options:

new ExpandClient({
    apiKey: 'xpnd_...',         // defaults to process.env.EXPAND_API_KEY
    baseUrl: 'https://api...',  // defaults to https://api.expand.ai
    timeoutMs: 60_000,          // total wall-clock budget incl. retries
    maxRetries: 2,              // retries transient failures; see "What gets retried"
})

Errors thrown:

• ExpandClientError — invalid options (missing API key, empty baseUrl)
• ExpandClientApiError — non-2xx HTTP response
• ExpandClientConnectionError — network failure
• ExpandClientTimeoutError — request timed out

Effect service

import { NodeRuntime } from '@effect/platform-node'
import { Effect } from 'effect'
import { ExpandService } from '@expandai/sdk'

const program = Effect.gen(function* () {
    const expand = yield* ExpandService
    const result = yield* expand.fetch({ url: 'https://example.com' })
    yield* Effect.log(result)
})

NodeRuntime.runMain(program.pipe(Effect.provide(ExpandService.layer({ apiKey: process.env.EXPAND_API_KEY }))))

Layer options (same shape as ExpandClientOptions):

ExpandService.layer({
    apiKey: 'xpnd_...',         // defaults to process.env.EXPAND_API_KEY
    baseUrl: 'https://api...',  // defaults to https://api.expand.ai
    timeoutMs: 60_000,          // total wall-clock budget incl. retries
    maxRetries: 2,              // retries transient failures; see "What gets retried"
})

Typed errors in the Effect channel:

• ExpandSdkError — invalid options (missing API key, empty baseUrl)
• ExpandApiError — non-2xx HTTP response
• ExpandConnectionError — network failure
• ExpandTimeoutError — request timed out

Local pages and files

In Node.js, pass { local: true } to fetch() or fetchJson() with a localhost HTTP URL or a file or folder path. The same option works on ExpandService.

const markdown = await expand.fetch({ url: 'http://localhost:3000' }, { local: true })
const page = await expand.fetchJson({ url: './dist' }, { local: true })

Local mode requires an API key and temporarily shares the selected content publicly. Saved Snapshots remain shareable afterward. Never enable it for URLs or paths supplied by untrusted users. See Local pages and files for setup, folder boundaries, supported inputs, and errors.

Search

search() finds the pages that answer a question when you have no URL yet. The same method works on ExpandService.

const { results } = await expand.search({ query: 'how do I rotate a Stripe API key', maxResults: 3 })

Send a query of up to 2,000 characters, with maxResults from 1 to 10, defaulting to 10. Each result has a url, title, excerpt, and score, best first. The response also has the search's id: getSearch(id) reads it back for free, without searching again. A search costs 1.5 credits. See Search.

Crawl and capture projections

Creation and run-command replays require a caller-persisted idempotency key. One-off crawls close permanently when done, failed, or canceled; create a new crawl to run again. Events carry their own resume offset.

Replaying a creation key returns the original ordered Crawl IDs, even if a member has since expired. Reading an expired member returns 404; replay never recreates it or starts a replacement run.

How a crawl finds pages

A crawl starts at each source URL and stays on that URL's registrable domain and its subdomains.

• maxDepth counts clicks from your URL. 0 is the URL alone. 1 adds the pages it links to and the pages in the site's sitemap. 2 adds the pages those link to, and so on. Pages found in the sitemap or by search count as one click, because they were not on your list. A kind: 'sitemap' source is the exception: the URLs it lists are yours, so they start at depth 0. The default, 50, is the whole site up to maxPages.
• include decides what ends up in the results, not where the crawler goes. To reach /p/** from the homepage, the crawler opens the pages that link there, such as / and /damen, and follows their links. It does not keep those pages, and they do not count toward maxPages.
• exclude keeps the crawler out. A matching page is never opened, so nothing behind it is found through it.
• sync.freshness is how often every page is checked for changes, for example 24h. How each check is made is up to the crawler.

changePolicy.ignore removes matching subtrees from change comparison; changePolicy.volatile replaces them with stable markers. Both leave saved page content unchanged. Selectors support static tags, classes, IDs, attributes, combinators and structural pseudo-classes such as :not, :has and :nth-child. Interaction states such as :hover, pseudo-elements, named namespace prefixes, nested :has and :nth-child(... of ...) are rejected. Matching includes captured shadow subtrees and uses case-sensitive class and ID values even on quirks-mode pages.

import { Expand } from '@expandai/sdk'

const expand = new Expand({ apiKey: process.env.EXPAND_API_KEY })
const [crawl] = await expand.crawls.create(
    { sources: ['https://example.com/products'], formats: ['markdown', 'stateJson'] },
    { idempotencyKey: 'catalog-import-2026-09' },
)
if (!crawl) throw new Error('Expected a Crawl')

for await (const event of crawl.events()) {
    if (event.type !== 'page.added' && event.type !== 'page.changed') continue
    const capture = expand.captures(event.captureId)
    const state = event.page.stateJson ?? await capture.stateJson()
    for (const evidence of state.evidence) {
        const body = evidence.body !== undefined ? evidence.body : (evidence.bodyUrl ? await expand.fetchJson(evidence.bodyUrl) : undefined)
        // Process body, then persist event.offset in your application.
    }
}

captures(id) also exposes get(), markdown(), html(), evidence(key) and playgroundUrl(). These read stored captures, not fresh pages. JSON body URLs must belong to the configured API origin and cannot redirect. Projection reads accept signal and timeoutMs; they do not perform automatic retries.

Catalog and version lists support individual items, pages, or one awaited page:

const list = crawl.pages({ limit: 100, status: 'live' })
const first = await list // { items, nextCursor? }
for await (const page of list.pages()) { /* page.items */ }
for await (const item of list) { /* item.id, item.url */ }
const bounded = await list.collect({ limit: 200 })

const versions = expand.pages('page-id').versions({ limit: 20 })
const diff = await expand.pages('page-id').diff({ from: 'capture-before', to: 'capture-after' })

A list instance caches its first page. Stop iteration with break, or cancel reads through signal. crawl.page(id) and crawl.pageByUrl(url) read the current catalog projection. Page-version lists use before or at for historical reads.

events({ from: '-1' }) replays retained history, including changes, removals, and failures. It does not rebuild the current catalog. For a full catalog import or recovery after CrawlOffsetExpiredError, use the pinned snapshot:

for await (const batch of crawl.snapshot().pages()) {
    // Apply batch.items, then persist batch.nextOffset.
    // When batch.upToDate is true, that offset is the boundary to resume with crawl.events({ from }).
}

snapshot({ from: savedSnapshotCursor }) resumes an unfinished snapshot. Snapshot cursors belong to /snapshot; an older bootstrap checkpoint submitted to /events returns 410. Event offsets are opaque and sortable: copy them exactly rather than converting them to numbers. One-off event and activity iterators stop at stream EOF. Recurring reads return when caught up unless live: true is set.

Event reads also expose bounded server pages. Awaiting an event read returns one page, including an empty caught-up page, and closes that read. Resume using its nextOffset. Event reads do not cache that first page across separate invocations.

Each non-live event request and control request honors the client's timeoutMs across its retries and maxRetries limit. Explicit live: true reads reconnect indefinitely on transient failures until canceled, returned from, or closed with the client. A server refusal marked retryable: false stops either mode. done() polls until completion or cancellation; its finite pollIntervalMs must be no greater than 2147483647, and values below 50 are clamped to 50ms.

Crawl API refusals extend ExpandClientApiError and ExpandClientError, with status, body, code, and retryable available. CrawlOffsetExpiredError remains the specific 410 error. Local validation errors extend ExpandClientError through CrawlClientError.

Effect users can call yield* expand.crawls.create(request, { idempotencyKey }) through ExpandService, then use status, update, cancel, or run on the returned resources. These control methods share the service's HTTP client, retries, and typed ExpandError channel. Event iterators, done(), and capture projections currently use the Promise interface; the Effect service does not yet mirror those helpers.

Because these reads are thenable, Promise.resolve, Promise.all, or returning one from an async function also starts that one-page read. Live page iteration exposes empty long-poll timeout pages; item iteration hides them.

const batch = await crawl.events({ from: savedOffset })
if (batch) {
    for (const event of batch.items) { /* process event */ }
    // Persist batch.nextOffset after processing the whole batch.
}
for await (const batch of crawl.events({ live: true }).pages()) {
    // batch.crawlId identifies the partition, including in merged reads.
}

Merged page reads take from: { [crawlId]: offset }, just like merged item reads. Each returned batch belongs to one Crawl and carries that Crawl's nextOffset; never reuse it for a different Crawl.

Batched fetches

batched kicks off a job and returns immediately with an id. Poll getBatched(id) until batchedStatus is no longer RUNNING or QUEUED.

Promise API

import { ExpandClient } from '@expandai/sdk'

const expand = new ExpandClient()

const run = await expand.batched({ urls: ['https://example.com'] })

let status = await expand.getBatched(run.id)
while (status.batchedStatus === 'RUNNING' || status.batchedStatus === 'QUEUED') {
    await new Promise((resolve) => setTimeout(resolve, 1000))
    status = await expand.getBatched(run.id)
}

console.log(status.results)

Successful results include Markdown at data.markdown and State JSON at data.json by default when extraction succeeds.

Each result has a persisted status (QUEUED, RUNNING, SUCCEEDED, FAILED, or CANCELLED) alongside its data.

Effect service

import { NodeRuntime } from '@effect/platform-node'
import { Duration, Effect, Schedule } from 'effect'
import { ExpandService, type GetBatchedResponse } from '@expandai/sdk'

const program = Effect.gen(function* () {
    const expand = yield* ExpandService
    const run = yield* expand.batched({ urls: ['https://example.com'] })
    const final = yield* expand.getBatched(run.id).pipe(
        Effect.repeat(
            Schedule.recurUntil(
                (status: GetBatchedResponse) => status.batchedStatus !== 'RUNNING' && status.batchedStatus !== 'QUEUED',
            ).pipe(Schedule.addDelay(() => Duration.seconds(1))),
        ),
    )
    yield* Effect.log(final.results)
})

NodeRuntime.runMain(program.pipe(Effect.provide(ExpandService.layer({ apiKey: process.env.EXPAND_API_KEY }))))

The SDK generates an idempotency key and reuses it for automatic retries when one is omitted. Supply a stable idempotencyKey to deduplicate separate calls or calls made after a process restart. Reusing a key with a different payload returns 409 BatchedIdempotencyConflict.

What gets retried

Not every 5xx is transient. Some of them are verdicts about the page you asked for — the browser committed its own error document, or the page could not be captured — and repeating the request re-runs a full capture to reach the same answer, on your timeout budget and your quota.

So the server states the answer rather than leaving it to a status-code rule. Error bodies whose status is ambiguous (429, 500, 502, 503, 504, 529) carry a retryable boolean, and the SDK follows it:

Because the verdict travels in the body, a newly-terminal error class needs no SDK upgrade — an unrecognized error that says retryable: false is still not retried.

When a response carries Retry-After, the next attempt waits at least that long — re-jittered, so a batch of clients rejected together does not come back in lockstep. timeoutMs still bounds the whole operation, retries and waits included.

License

Apache 2.0