Web scraper

This pack gives an agent page scraping, structured extraction and named YAML recipe tools. It uses fetch, optional playwright-core browser fallbacks, a regex selector engine and yaml for recipe loading.

Installation

npm install @framers/agentos-ext-web-scraper

Peer dependency: @framers/agentos with range >=0.10.40.

Usage

import { AgentOS } from '@framers/agentos';

const agentos = await AgentOS.create({
  extensionManifest: {
    packs: [
      {
        package: '@framers/agentos-ext-web-scraper',
        options: {
          proxyUrl: process.env.WEB_SCRAPER_PROXY_URL,
          proxyList: process.env.WEB_SCRAPER_PROXY_LIST,
        },
      },
    ],
  },
});

Configuration

Proxy settings resolve from truthy options, then context.getSecret, then the environment. proxyList accepts a comma-separated string or an array of strings and takes precedence over proxyUrl. The service rotates through the list; per-request options.proxy overrides it. logger.info receives activation and deactivation messages.

Tools

web_scrape

web_scrape_recipe

Notes

Both tools declare capability:web_scrape. The service tries plain fetch, headless Playwright, stealth Playwright, then a text-fetch fallback. javascript: true starts at the browser tier unless maxTier caps it lower. A cached successful tier can raise the starting tier. Fetch requests and browser navigation or selector waits default to 30000 ms per operation, rather than a total deadline.

Browser fallbacks dynamically import playwright-core and launch Chromium. If the import or launch fails, the service continues to the next allowed tier. playwright-core is not listed in the package's dependencies. Proxies are passed to Playwright launch options; the fetch paths do not apply them. Each browser attempt closes its browser afterward.

The text fallback rejects content shorter than 200 characters and recognized Cloudflare challenge pages. With extract, it returns text and _llmExtractionRequired: true; the calling runtime performs any LLM extraction. Browser paths also reject recognized challenge pages. Plain fetch accepts successful HTTP responses without this challenge check. Failure across all allowed tiers returns success: false and an error message.

Extraction supports tag, .class, #id, tag.class and a @attr suffix. It uses regex matching rather than a full CSS parser. Unsupported or unmatched field selectors return empty strings. Empty extraction results do not trigger fallback escalation.

Rate limiting uses random delays between minDelayMs and maxDelayMs for repeated requests to a domain. Successful tiers are cached per domain for 30 minutes, with a maximum of 500 entries.

Activation loads YAML recipes from the package's recipes/ directory, then ~/.wunderland/scraper-recipes/; user recipes override matching names. The bundled recipe names are generic-article and letterboxd-movie. The engine looks up loaded names and does not load a path supplied as recipe. Missing recipes return an error. Invalid recipe files produce console warnings and are skipped.

Recipe steps run sequentially and interpolate {{paramName}}, {{steps.stepName.data.field}} and {{steps.stepName.items[N].field}} in URLs and selectors. Parameters prefixed with = in a recipe have defaults; missing values otherwise become empty strings. Interpolation does not URL-encode values. Failed steps stop execution and return partial data. A step's options.maxItems caps list results. The bundled Letterboxd recipe uses descendant selectors that the regex extraction engine does not implement.

License

Apache-2.0