@gscdump/engine

中文文档
7.15.0发布于 2026/11/16MITTypeScriptES Module构建来源已验证近 30 天下载 1万

Append-only Parquet/DuckDB storage engine + planner + adapters for the gscdump pipeline. Node + edge runtimes; opt-in heavy peers.

原始使用文档
文档目录5 节
  1. Install
  2. Subpath exports
  3. Stability
  4. Related
  5. License

@gscdump/engine

npm version npm downloads license

Store Search Console data in Parquet files and query it with DuckDB. This package also provides Source contracts, SQL resolution, and runtime adapters.

Choose an export for your runtime. Node filesystem and DuckDB helpers live on dedicated subpaths.

Install

npm install @gscdump/engine

@duckdb/node-api is an optional peer for the Node runtime. The CLI installs this peer. Browser consumers use @gscdump/engine-duckdb-wasm. The package includes hyparquet and hyparquet-writer for JavaScript Parquet encoding and decoding.

The /hyparquet encoders accept optional fileMetadata: Record<string, string>: use encodeRowsToParquet(table, rows, { fileMetadata }), or include it in the options for encodeRowsToParquetFlex. Metadata is stored in the file footer, not as fact columns, including for empty files. Keys are sorted for deterministic encoding; omitting metadata or passing an empty object preserves the existing bytes. A publication coordinator can supply a unique identifier so identical rows from different publications produce different bytes. It must retain and replay the exact encoded bytes for retries; this option does not provide publication or recovery coordination itself.

For a bounded, already ordered source, /hyparquet also exports encodeParquetRowBatches(batches, options, write). It consumes sync or async batches, coerces the declared columns with the same physical types as the array encoder, and emits one Parquet file through an awaited byte callback. It does not sort, deduplicate, retry, or publish an object. The callback borrows its bytes until its promise settles; copy a chunk if retaining it. The caller must abort/discard the entire destination on any error, including an error while writing the footer. Reopening the same immutable source with identical options produces identical bytes regardless of producer batch boundaries. This supports a counting/digest pass followed by a separately coordinated fixed-length upload, without the codec owning transport or commit semantics.

The streaming encoder normally refuses when retained footer/index descriptors exceed limits.maxRetainedMetadataBytes (8 MiB by default). For wide values, callers can explicitly choose metadataOverflow: 'omit-statistics'. On the first overflow it omits column-chunk statistics and ColumnIndexes throughout that file, including earlier groups. OffsetIndexes, encoding descriptors, schema, row counts, data/dictionary pages and explicit fileMetadata remain unchanged. Required descriptors must still fit the same budget, and the group/footer output-buffer limit still applies. Missing statistics mean unknown bounds, so readers may scan more pages or groups; they do not mean empty data. If no overflow occurs, this option preserves the normal bytes. It bounds retained optional descriptors, not their transient construction or total heap. The array encoders and all omitted-option behavior remain unchanged.

decodeParquetRowGroups({ byteLength, slice }, options) reads exact byte ranges and by default yields one physical row group at a time. The caller must pin every range to the same immutable object or ETag. It preserves the existing date, column projection, row-window and filter behavior. An absent, corrupt, oversized or incompletely read file throws; only valid empty Parquet yields no rows.

The range reader alone accepts an optional indexedRowWindow policy:

decodeParquetRowGroups(pinnedObject, {
  indexedRowWindow: {
    targetRows: 1000,
    requiredIndexedColumns: ['query', 'url'],
    rangeCacheBytes: 2 * 1024 * 1024,
  },
})

Each group uses windows only when all designated columns are present, selected, have usable offset-index descriptors, and have no dictionary encoding. The schema must contain only flat scalar/UTF8/date fields; binary, nested and other unsupported schemas retain the original group read. Ineligible groups use the original uncached path, so targetRows is a target rather than a hard batch limit. Include every potentially large column in requiredIndexedColumns. All existing physical-group admission limits still apply.

Before emitting an eligible group's first window, the reader checks index structure for projected columns, or all indexed columns when a filter may need unprojected fields: row zero, increasing row positions, ordered nonoverlapping page ranges and physical column bounds. A failure throws without falling back after partial output. This assumes metadata row offsets agree with physical page contents; it does not independently verify page-header row counts or detect every deliberately consistent but false index.

The optional range cache defaults to zero, allows at most 2 MiB of payload and 128 entries, and lives only through one eligible group. It is cleared on completion, error or iterator return; another pass rereads the pinned object. It does not cache decoded rows or provide publication/ETag coordination. Windows can otherwise reread overlapping pages and substantially increase range I/O. They still decode whole boundary pages, and dictionaries in other columns can still allocate in full. These options do not bound decompressed page sizes or the total heap; the unchanged fallback also retains its existing memory costs. Buffered decoders, encoders and all omitted-option behavior remain unchanged.

The range reader can also opt into limits.maxReadAndDecompressBytes. This is a cumulative buffer-work budget for each physical group, shared across all its indexed windows and concurrent column reads. It charges requested source-range bytes before dispatch and SNAPPY output bytes before allocating the fixed-size output buffer. Cache hits do not allocate another range, while repeated decompression still consumes budget. Group indexes are included; the initial footer has its separate metadata limit. Raw/uncompressed pages and V2 levels are covered by the range charge, so compressed data is deliberately counted twice. Earlier charges are never refunded, including after a failed read. Failure or iterator return closes the budget, preventing new work and rejecting late range results. It cannot physically cancel already dispatched reads: the range source interface has no cancellation hook.

ZSTD decompression refuses under this option: the current allocating decoder can allocate from its frame header before checking the Parquet output size. Without the option, ZSTD and all existing decoding behavior remain unchanged. The budget does not measure JavaScript strings, expanded values, row containers, internal prefetch or dictionary/page stitching copies, codec working state or caller-retained batches. It is not a total heap bound or protection against all malformed Parquet contents. It can reject valid files that previously decoded, particularly old files whose writer understated uncompressed footer sizes. Callers must preserve those files and choose their own explicit retry/repair policy; refusal is never empty data.

decodeParquetRowGroups(source, { boundedValues: {} }) selects a separate strict value-cursor path. It supports flat REQUIRED/OPTIONAL scalar columns, explicitly annotated UTF8/STRING, PLAIN and dictionary data pages (V1/V2), and SNAPPY or UNCOMPRESSED data. Projection, physical row ranges, nulls, date normalization and row order are preserved. Nested/binary/other logical types, other codecs or value encodings, BOOLEAN dictionaries, filters and indexedRowWindow combinations refuse; this mode never silently falls back to the eager reader. All selected page headers are inspected before the first output to detect unsupported later pages. Corrupt value bodies or over-limit individual values may fail after an earlier batch; consumers must finish validating their entire input before publishing a replacement. Explicit row bounds in this strict mode must be nonnegative safe integers, with positive Infinity also accepted for an open-ended rowEnd. Omitted bounds use the full input, and reversed or zero-length ranges are empty.

The cursor retains at most one current data page and one encoded dictionary per selected column. UTF8 strings, definition levels and dictionary IDs are consumed incrementally. Dictionaries retain encoded bytes and a four-byte offset per entry, rather than an array of decoded strings. Final bit-packed padding is validated but not expanded. These options have independent meanings:

boundedValues optionDefaultAdmission rule
targetRows1000Hard emitted batch row limit
maxBatchBytes1 MiBBefore materialization: 128 per row, 16 per cell slot, 64 per cell plus twice UTF8 byte length or 32 for a fixed scalar
maxValueBytes256 KiBEncoded bytes of one string, before decoding
maxActiveBytes24 MiBPer group: sum of each column's largest compressed + uncompressed data page, dictionary compressed + uncompressed bytes, dictionary offset tables, plus one maximum header probe
maxPages16384Retained page-descriptor count across the selected input
maxHeaderBytes64 KiBHeader probe cap; starts at 512 bytes and grows only on truncation
maxDictionaryEntries100000Entry count for each encoded dictionary

Active-buffer admission uses real page headers, not the often understated uncompressed sizes in legacy writer footers. Page transitions release the old cursor before reading/decompressing its successor; columns load sequentially. Batch bytes are a conservative value-materialization charge, not a measurement of V8 object overhead or an isolate heap guarantee. The footer (separately bounded), page-descriptor objects, fixed cursor/schema state, caller-retained batches, codec/runtime overhead and concurrent application work remain separate. Setting larger limits deliberately permits a larger working set. A single page still decompresses in full, and a consumer that retains all emitted batches removes the benefit. This mode does not make arbitrary malicious files safe.

The existing maxReadAndDecompressBytes option also applies: a group's header probes, physical reads and SNAPPY output share one cumulative budget. Headers are scanned once; immutable/pinned object bytes are required across that scan and later reads. Iterator return/error releases cursor references; already dispatched source reads cannot be physically aborted. Defaults, the buffered decoder and indexed-window eligibility/fallback are unchanged.

Both APIs have configurable admission limits. The encoder bounds producer batch rows/value bytes, output group rows/value bytes, retained descriptor counts and scalar bytes (including statistics), and the current encoded group/footer buffer. The decoder checks footer size, descriptor count, declared group rows/uncompressed bytes, and every range request. The default decoder accepts the existing encoder's 25,000-row groups. These limits do not establish a Worker heap guarantee: caller-retained rows, codec transient allocations, footer descriptors, and concurrent requests still use memory. Metadata limits are not protection against malicious Parquet contents. The existing array APIs and their default bytes are unchanged.

Subpath exports

SubpathPurpose
@gscdump/engineStorage runtime: createStorageEngine, codec/executor factories, storage contracts.
@gscdump/engine/contractsStorageEngine, Row, TableName, WriteCtx, ManifestEntry, ...
@gscdump/engine/schemaSCHEMAS, allTables, inferTable, column metadata.
@gscdump/engine/plannerresolveToSQL, enumeratePartitions, partition planning.
@gscdump/engine/ingestGSC row → storage row helpers (createRowAccumulator, transformGscRow).
@gscdump/engine/ingest-accumulatorStateful ingest flushing behind injected storage and recovery hooks.
@gscdump/engine/sqlSQL literal binding helpers (bindLiterals, formatLiteral).
@gscdump/engine/sql-fragmentsReusable SQL fragments shared across analyzers.
@gscdump/engine/rollupsPre-aggregated rollup contracts and helpers.
@gscdump/engine/entitiesParquet-backed entity stores and their record contracts.
@gscdump/engine/entity-keysPure entity object-key and URL-hash helpers, with no Parquet runtime.
@gscdump/engine/resolverDialect-neutral SQL composition: ResolverAdapter, pgResolverAdapter, resolveToSQL.
@gscdump/engine/scopeMulti-tenant scope predicates.
@gscdump/engine/arrowApache Arrow utilities for engine result conversion.
@gscdump/engine/nodeNode-only DuckDB handle.
@gscdump/engine/filesystemNode-only DataSource and ManifestStore adapters.
@gscdump/engine/hyparquetPure-JS ParquetCodec.
@gscdump/engine/r2Cloudflare R2 DataSource and createR2ManifestStore (structurally typed against R2Bucket).
@gscdump/engine/icebergEdge-safe GSC dataset schemas, catalog wrappers, and append sink. Generic Iceberg APIs live in @gscdump/lakehouse.
@gscdump/engine/sink-nodeNode-only Iceberg overwrite/delete recovery writer.
@gscdump/engine/sourceSource contracts and factories.
@gscdump/engine/analyzerAnalyzer contracts, registry construction, and dispatch.
@gscdump/engine/reportReport definitions and contracts.
@gscdump/engine/periodDate windows and comparison ranges.
@gscdump/engine/analysis-typesAnalysis parameters and results.
@gscdump/engine/analysis-rangeAnalysis date-range resolution.
@gscdump/engine/errorsTagged Engine failures.
@gscdump/engine/profileQuery timing and profiling helpers.
@gscdump/engine/sync-configSync limits and table/search-type configuration.
@gscdump/engine/vendor/hysnappyWorker-compatible compression alias.

Stability

SurfaceStability
createStorageEngine and storage contracts (StorageEngine, Row, WriteCtx, ...)Public
Adapters (/node, /filesystem, /hyparquet, /r2)Public
Planner (resolveToSQL, enumeratePartitions)Public
Schema (SCHEMAS, allTables, ...)Public
Files outside the declared package exportsPrivate

License

MIT

版本与发布记录

@gscdump/engine 共发布 268 个版本,首次发布于 2026/04/27;过去 12 个月发布了 268 个版本。最新版本附带 npm provenance 构建来源证明,可追溯到公开的 CI 构建记录。

版本发布日期距今
7.15.0latest2026/11/16今天
7.14.02026/11/16今天
7.13.22026/11/151 天前
7.13.12026/11/152 天前
7.13.02026/11/124 天前
7.12.12026/11/116 天前

关于 @gscdump/engine 的常见问题

@gscdump/engine 是什么?

Append-only Parquet/DuckDB storage engine + planner + adapters for the gscdump pipeline. Node + edge runtimes; opt-in heavy peers.

如何安装 @gscdump/engine?

在项目目录运行 npm install @gscdump/engine,或使用 pnpm add / yarn add / bun add。

@gscdump/engine 支持 TypeScript 吗?

支持。包内自带 TypeScript 类型声明(package.json 声明了 types 入口),安装后即可获得类型提示,无需再安装 @types 包。

@gscdump/engine 支持 ESM 还是 CommonJS?

以 ES Module 形式发布,推荐使用 import 引入。在 CommonJS 项目中,Node.js 20.19+ / 22.12+ 可以直接 require() 不含顶层 await 的 ES Module,更早的版本需要改用动态 import()。

@gscdump/engine 需要什么版本的 Node.js?

package.json 的 engines 字段要求 Node.js >=22.13.0。版本不符时 npm 默认只给出警告,开启 engine-strict 后会拒绝安装。

@gscdump/engine 可以商用吗?

@gscdump/engine 使用 MIT 许可。宽松许可:允许商业使用、修改和再分发,通常只需在分发时保留原版权与许可声明。以上为概括,不构成法律意见,请以许可原文为准。

@gscdump/engine 的下载量有多少?

根据 npm 官方下载统计,@gscdump/engine 在 2026/10/17 – 2026/11/15 这 30 天内被下载 10,417 次,日均约 347 次。下载量会把 CI 构建、镜像同步和作为其他包依赖的安装一并计入,适合横向比较热度,不能直接等同于使用人数。

@gscdump/engine 还在维护吗?

最新版本 7.15.0 发布于 2026/11/16(今天);至今共发布 268 个版本,其中过去 12 个月发布了 268 个;首次发布于 2026/04/27。发布频率只是参考信号:功能稳定的工具可能很少需要更新,建议结合源码仓库的 issue 和提交记录判断。