@anycrawl/js-sdk (英语)

AnyCrawl API的轻量级ESM JavaScript/TypeScript客户端.

所需资源

• 节点.js 18+(建议标点20-匹配CI)
• 无害环境管理模块

安装

pnpm add @anycrawl/js-sdk

快速启动

import { AnyCrawlClient } from "@anycrawl/js-sdk";

// Prefer loading from env in your app framework
const client = new AnyCrawlClient(process.env.ANYCRAWL_API_KEY || "<YOUR_API_KEY>");
// Base URL defaults to https://api.anycrawl.dev, but you can override:
// const client = new AnyCrawlClient(process.env.ANYCRAWL_API_KEY!, "https://api.anycrawl.dev");

// Health
await client.healthCheck(); // -> { status: "ok" }

// Scrape (engine defaults to "auto" — automatically picks the best engine)
const scrape = await client.scrape({
    url: "https://example.com",
    formats: ["markdown"],
});

// Crawl (async job)
const job = await client.createCrawl({
    url: "https://anycrawl.dev",
    max_depth: 3,
    strategy: "same-domain",
    limit: 50,
    scrape_options: { formats: ["markdown"] },
});
const status = await client.getCrawlStatus(job.job_id);
const page1 = await client.getCrawlResults(job.job_id, 0);

// Cancel if needed
// await client.cancelCrawl(job.job_id);

// Blocking helper (waits until the crawl finishes and aggregates all pages)
const aggregated = await client.crawl(
    {
        url: "https://anycrawl.dev",
        max_depth: 3,
        strategy: "same-domain",
        limit: 50,
    },
    2, // poll interval (seconds)
    10 * 60_000 // optional timeout (ms)
);
// aggregated.data contains all results

// Map (discover all URLs from a website)
const mapResult = await client.map({
    url: "https://anycrawl.dev",
    limit: 100,
    include_subdomains: false,
    ignore_sitemap: false,
});
// mapResult.links contains all discovered URLs

// Search (optionally enrich with scraping)
const results = await client.search({
    query: "OpenAI ChatGPT",
    engine: "google",
    pages: 1,
    limit: 10,
    scrape_options: { engine: "cheerio", formats: ["markdown"] },
});

使用详情

客户端

new AnyCrawlClient(apiKey: string, baseUrl = "https://api.anycrawl.dev", onAuthFailure?: () => void)

• apiKey: API呼叫的熊克指使.
• baseUrl:如果自我宿主,则征服.
• onAuthFailure: 401/403上传. 用于触发退出或刷新.

您也可以设置 LOG_LEVEL 控制内部日志debug, (中文(简体) ), info, (中文(简体) ), warn, (中文(简体) ), error) (中文(简体) ).

刮去(输入)

import type { Engine, ScrapeFormat } from "@anycrawl/js-sdk";

const engine: Engine = "auto"; // or "cheerio", "playwright", "puppeteer"
const formats: ScrapeFormat[] = ["markdown", "html"];

await client.scrape({
    url: "https://example.com",
    engine, // optional — defaults to "auto"
    // Optional:
    template_id: "my-template",
    variables: { key: "value" },
    proxy: "http://user:pass@host:port",
    formats,
    timeout: 60_000,
    retry: true,
    wait_for: 3000,
    wait_for_selector: ".content",
    only_main_content: true,
    include_tags: ["article", "main"],
    exclude_tags: ["nav", "footer"],
    json_options: { user_prompt: "Extract title", schema: { type: "object" } },
    extract_source: "markdown", // or "html"
    ocr_options: false,
    max_age: 86_400_000,
    store_in_cache: true,
});

返回带有内容的成功对象或带有失败 error。 。 。 .

创建 Crawl( 输入), 获取 CrawlStatus( jobId), 获取 CrawlResults( jobId, 跳过?), 取消 Crawl( 工作) (d) 国家

const job = await client.createCrawl({
    url: "https://site.com/docs",
    engine: "playwright", // optional — defaults to "auto"
    template_id: "my-template",
    variables: { section: "api" },
    max_depth: 5,
    strategy: "same-domain",
    limit: 100,
    include_paths: ["/docs/*"],
    exclude_paths: ["/admin/*"],
    scrape_paths: ["/docs/*"], // Only scrape content from these paths
    scrape_options: { formats: ["markdown"] },
});

const status = await client.getCrawlStatus(job.job_id);
const page = await client.getCrawlResults(job.job_id, 0);
// await client.cancelCrawl(job.job_id);

爬行(输入,民调Intervalseconds = 2,超时?)

创造爬行、民意调查状态直至完成, 扔出失败的工作或超时; 取消时返回部分汇总数据 .

// Type signature
async function crawl(
    input: CrawlRequest,
    pollIntervalSeconds?: number, // default 2s
    timeoutMs?: number // optional
): Promise<CrawlAndWaitResult>;

// Example
try {
    const aggregated = await client.crawl(
        {
            url: "https://anycrawl.dev",
            max_depth: 3,
            strategy: "same-domain",
            limit: 50,
            scrape_options: { formats: ["markdown"] },
        },
        3, // poll every 3s
        5 * 60_000 // timeout after 5 minutes
    );
    console.log(aggregated.total, aggregated.completed);
    console.log(aggregated.data.length, "pages aggregated");
} catch (err) {
    // Handles API/network/auth errors, job failed, or timeout
    console.error(err);
}

• 回返情况: CrawlAndWaitResult 与 job_id, (中文(简体) ), status, (中文(简体) ), total, (中文(简体) ), completed, (中文(简体) ), creditsUsed合计数 data。 。 。 .
• 投票状态:等待 completed; 弹出 failed; 当 cancelled。 。 。 .
• 使用 getCrawlStatus编号 :getCrawlResults 如果您喜欢手动页码/ 进度.

创建BatchScrape(输入),获取BatchScrapeStatus(工作ID),获取BatchScrapeResults(工作ID,跳过吗),取消BatchScrape(工作) (d) 国家

使用一个共享的 raise 选项集( 无链接发现) , 将许多已知的 URL 在一个同步任务中搜索 。 每被成功刮去的 URL 收取信用;失败的 URL 不收取信用 .

const job = await client.createBatchScrape({
    urls: ["https://example.com/a", "https://example.com/b"],
    engine: "auto", // optional — same scrape options as scrape()
    formats: ["markdown"],
    // Or use a scrape template applied to every URL:
    // template_id: "tpl_product_page",
    // variables: { currency: "USD" },
    ignore_invalid_urls: true,
});

const status = await client.getBatchScrapeStatus(job.job_id);
const page = await client.getBatchScrapeResults(job.job_id, 0);
// await client.cancelBatchScrape(job.job_id);

批次Scrape(输入,民意调查Intervalseconds = 2,超时?)

创建分批擦拭的便捷包装, 民意调查状态直至完成, 并返回所有 URL 的汇总结果 。 扔出失败的工作或超时; 取消时返回部分汇总数据 .

// Type signature
async function batchScrape(
    input: BatchScrapeRequest,
    pollIntervalSeconds?: number, // default 2s
    timeoutMs?: number // optional
): Promise<BatchScrapeAndWaitResult>;

// Example
const result = await client.batchScrape({
    urls: ["https://example.com/a", "https://example.com/b"],
    formats: ["markdown"],
});
console.log(result.completed, result.failed, result.data.length);

• 回返情况: BatchScrapeAndWaitResult 与 job_id, (中文(简体) ), status, (中文(简体) ), total, (中文(简体) ), completed, (中文(简体) ), failed, (中文(简体) ), creditsUsed合计数 data (每件物品都有自己的 url) (中文(简体) ).
• 投票状态:等待 completed; 弹出 failed; 当 cancelled。 。 。 .
• 使用 getBatchScrapeStatus编号 :getBatchScrapeResults 如果您喜欢手动页码/ 进度.

搜索( 输入)

await client.search({
    query: "best js tutorials",
    engine: "google",
    limit: 20,
    offset: 0,
    pages: 2,
    lang: "en",
    country: "US",
    timeRange: "week",
    sources: "web",
    template_id: "search-template",
    variables: {},
    scrape_options: { engine: "cheerio", formats: ["markdown"] },
    safe_search: 1,
});

映射( 输入)

await client.map({
    url: "https://example.com",
    limit: 100,
    include_subdomains: false,
    ignore_sitemap: false,
});

使用多个来源从网站上发现所有 URL :

• 分解( siteap.xml, 机器人. txt)
• 搜索引擎结果(网站:运营商)
• 页面链接提取( HTML) 标签)

回返 MapResult 与 links 包含 { url, title?, description? } 对象.

模板( ref)

通过自己的专用端点调用模板而不是通过 template_id 在刮伤/搜索/爬行体. ref 是模板的虚空弹片(首选)或它的 templateId返回绑定到 ref 与 execute 财务报告和审定财务报表 spec。 。 。 .

const t = client.template("content-extractor"); // slug or templateId

// Discover the call-spec (free — never billed). Returns TemplateSpec.
const spec = await t.spec();
// spec.template_type -> "scrape" | "search" | "crawl"
// spec.variables, spec.pricing, spec.allowed_domains, spec.endpoint

// Execute. Body accepts only url / query / variables — every other option
// comes from the template. Return type depends on the template type.
const result = await t.execute({
    url: "https://example.com", // use `query` for search templates
    variables: { category: "tech" },
});

结果是: execute() 实值 TemplateExecuteResult,受模板类型歧视:

• 刮伤 → ScrapeResult (同步)
• 搜索 → SearchResult[] (同步)
• 爬动 → CrawlJobResponse (单位:千美元){ job_id, status, message }——同心协力;与 client.getCrawlStatus(job_id) 编号 : client.getCrawlResults(job_id) (或 client.crawl(...)(中文(简体) )

// Crawl template: async job, then poll
const job = await client.template("docs-crawler").execute({ url: "https://docs.example.com" });
if ("job_id" in job) {
    const status = await client.getCrawlStatus(job.job_id);
    const page = await client.getCrawlResults(job.job_id, 0);
}

错误反映了基本授权调用加模板特有案例: 404 (未找到或无法访问), 403 (URL不被模板域政策所允许),, 400 (缺失/无效变量)。 见 处理错误。 。 。 .

计划任务

// Create
const created = await client.createScheduledTask({
    name: "Daily scrape",
    cron_expression: "0 9 * * *",
    timezone: "Asia/Shanghai",
    task_type: "scrape",
    task_payload: { url: "https://example.com", engine: "cheerio", formats: ["markdown"] },
    concurrency_mode: "skip",
    max_executions_per_day: 1,
});
// created.task_id, created.next_execution_at

// List, get, update, delete
const tasks = await client.listScheduledTasks();
const task = await client.getScheduledTask(created.task_id);
await client.updateScheduledTask(created.task_id, { cron_expression: "0 10 * * *" });
await client.pauseScheduledTask(created.task_id, "Maintenance");
await client.resumeScheduledTask(created.task_id);
await client.deleteScheduledTask(created.task_id);

// Executions
const { data } = await client.getScheduledTaskExecutions(created.task_id, { limit: 10, offset: 0 });
await client.cancelScheduledTaskExecution(created.task_id, executionId);

网络呼号

// Create
const webhook = await client.createWebhook({
    name: "My webhook",
    webhook_url: "https://your-server.com/webhook",
    event_types: ["scrape.completed", "crawl.completed"],
    scope: "all",
});
// webhook.webhook_id, webhook.secret (save secret - shown only once)

// List, get, update, delete
const webhooks = await client.listWebhooks();
const w = await client.getWebhook(webhook.webhook_id);
await client.updateWebhook(webhook.webhook_id, { event_types: ["scrape.completed"] });
await client.deleteWebhook(webhook.webhook_id);

// Deliveries, test, activate/deactivate
const { data } = await client.getWebhookDeliveries(webhook.webhook_id, { limit: 20, status: "failed" });
await client.testWebhook(webhook.webhook_id);
await client.activateWebhook(webhook.webhook_id);
await client.deactivateWebhook(webhook.webhook_id);
await client.replayWebhookDelivery(webhook.webhook_id, deliveryId);

// Supported event types
const events = await client.getWebhookEvents();
// events.event_types, events.categories

监视器

// Create a webpage change monitor
const monitor = await client.createMonitor({
    name: "Docs watch",
    monitor_type: "webpage",
    cron_expression: "0 * * * *",
    targets: [{ url: "https://example.com/changelog", engine: "auto" }],
    diff_options: { only_main_content: true },
    notify_options: { channels: ["webhook"], only_meaningful: true },
});
// monitor.monitor_id, monitor.scheduled_task_id, monitor.next_execution_at

// List, get, update, delete
const monitors = await client.listMonitors();
const m = await client.getMonitor(monitor.monitor_id);
await client.updateMonitor(monitor.monitor_id, { cron_expression: "0 */2 * * *" });
await client.deleteMonitor(monitor.monitor_id);

// Lifecycle and history
await client.pauseMonitor(monitor.monitor_id);
await client.resumeMonitor(monitor.monitor_id);
await client.runMonitor(monitor.monitor_id);
const snapshots = await client.getMonitorSnapshots(monitor.monitor_id, { limit: 10 });
const changes = await client.getMonitorChanges(monitor.monitor_id, { limit: 10 });
await client.getMonitorChange(monitor.monitor_id, changeId);

注释:

• 引擎默认为 auto 省略时——服务器自动选择最佳引擎(cheerio 静态页面, playwright 《联合来文》各页.
• Scrape 选项位于顶层; craw 接受嵌入 scrape_options 只有; 顶级只允许爬行策略字段和可选 retry。 。 。 .
• 搜索支持可选 scrape_options; 未提供时 engine,它被省略(没有每次结果刮出浓缩;使用浓缩时API默认自动).

运行 E2E 测试

现场整合测试击中了真正的AnyCrawl API并消耗了信用. 他们是 没有 包含在默认中 pnpm test 或者说 pnpm test:coverage 运行.

先决条件

• 一个有效的 ANYCRAWL_API_KEY
• 设定 ANYCRAWL_RUN_LIVE=1 选择加入( 双门防止在 CI 中意外执行)
• (备选) ANYCRAWL_BASE_URL 对自办案件进行测试(违约 https://api.anycrawl.dev(中文(简体) )

运行

# Inline env vars
ANYCRAWL_API_KEY=sk-xxx ANYCRAWL_RUN_LIVE=1 pnpm --filter @anycrawl/js-sdk test:e2e

# Or configure in root .env and run:
pnpm --filter @anycrawl/js-sdk test:e2e

测试套房按信用成本分级排列:

处理错误

所有方法抛出标准 Error 可读信件。 实例:

• 认证错误 : Authentication failed: <message> (401/403) - 触发器 onAuthFailure 如果提供
• 所需付款: Payment required: <message>. current_credits=<n> (402起)
• API 错误 : API Error <status>: <message>
• 网络问题: Network error: Unable to reach AnyCrawl API
• 其他请求错误 : Request error: <message>

接听电话 try/catch 用于处理应用程序中的错误.

注释:

• 一些端点返回 HTTP 2xx 与 { success: false, error?: string }。SDK将这些转换成 Request error: <message> (例如, Request error: Scraping failed) (中文(简体) ).
• 401和403均被视为认证失败,并将援引 onAuthFailure 当设置.
• 402年,当答复包括: current_credits,则信息具有上述特殊性;否则信息会回到 API Error 402: <message>。 。 。 .

API 表面

核心

• healthCheck(): Promise<{ status: string }>
• setAuthFailureCallback(cb: () => void): void
• scrape(input: ScrapeRequest): Promise<ScrapeResult>
• createCrawl(input: CrawlRequest): Promise<CrawlJobResponse>
• getCrawlStatus(jobId: string): Promise<CrawlStatusResponse>
• getCrawlResults(jobId: string, skip?: number): Promise<CrawlResultsResponse>
• crawl(input: CrawlRequest, pollIntervalSeconds?: number, timeoutMs?: number): Promise<CrawlAndWaitResult>
• cancelCrawl(jobId: string): Promise<{ job_id: string; status: string }>
• search(input: SearchRequest): Promise<SearchResult[]>
• map(input: MapRequest): Promise<MapResult>
• template(ref: string): { execute(input?: TemplateExecuteRequest): Promise<TemplateExecuteResult>; spec(): Promise<TemplateSpec> }

计划任务

• createScheduledTask(input): Promise<ScheduledTaskCreateResponse>
• listScheduledTasks(): Promise<ScheduledTask[]>
• getScheduledTask(taskId): Promise<ScheduledTask>
• updateScheduledTask(taskId, input): Promise<ScheduledTask>
• pauseScheduledTask(taskId, reason?): Promise<void>
• resumeScheduledTask(taskId): Promise<void>
• deleteScheduledTask(taskId): Promise<void>
• getScheduledTaskExecutions(taskId, params?): Promise<ScheduledTaskExecutionsResponse>
• cancelScheduledTaskExecution(taskId, executionId): Promise<void>

网络呼号

• createWebhook(input): Promise<WebhookCreateResponse>
• listWebhooks(): Promise<Webhook[]>
• getWebhook(webhookId): Promise<Webhook>
• updateWebhook(webhookId, input): Promise<void>
• deleteWebhook(webhookId): Promise<void>
• getWebhookDeliveries(webhookId, params?): Promise<WebhookDeliveriesResponse>
• testWebhook(webhookId): Promise<void>
• activateWebhook(webhookId): Promise<void>
• deactivateWebhook(webhookId): Promise<void>
• replayWebhookDelivery(webhookId, deliveryId): Promise<void>
• getWebhookEvents(): Promise<WebhookEventsResponse>

监视器 (所需经费) @anycrawl/js-sdk 0.0.6+) (中文(简体) )

• createMonitor(input): Promise<MonitorCreateResponse>
• listMonitors(): Promise<Monitor[]>
• getMonitor(monitorId): Promise<Monitor>
• updateMonitor(monitorId, input): Promise<Monitor>
• deleteMonitor(monitorId): Promise<void>
• pauseMonitor(monitorId): Promise<void>
• resumeMonitor(monitorId): Promise<void>
• runMonitor(monitorId): Promise<void>
• getMonitorSnapshots(monitorId, params?): Promise<MonitorSnapshot[]>
• getMonitorChanges(monitorId, params?): Promise<MonitorChange[]>
• getMonitorChange(monitorId, changeId): Promise<MonitorChange>

类型定义导出自 @anycrawl/js-sdk 用于 TypeScript 用户.