@gscdump/lakehouse

中文文档
7.15.0发布于 2026/11/16MITTypeScriptES Module零依赖构建来源已验证近 30 天下载 1万

Dataset-agnostic Iceberg lakehouse layer + dataset registry for R2 Data Catalog producers (ADR-0021).

原始使用文档
文档目录7 节
  1. Install
  2. Public surfaces
  3. Opt-in manifest read limits
  4. Append retries
  5. Manifest merging
  6. Table restoration
  7. License

@gscdump/lakehouse

Dataset-agnostic Iceberg catalog, schema, and dataset-registry primitives for R2 Data Catalog producers.

Install

npm install @gscdump/lakehouse

Node.js 22 or newer is required.

Public surfaces

  • @gscdump/lakehouse: connect to a catalog, define datasets, derive table specifications, resolve data files, and use bigint-safe serialization.
  • @gscdump/lakehouse/bigint: bigint-safe JSON helpers without catalog or Node dependencies.
  • @gscdump/lakehouse/schema: lightweight shared schema constants.
  • @gscdump/lakehouse/maintenance: stable commit classification, manifest merging, and orphan cleanup workflows.
  • @gscdump/lakehouse/provisioning: adopt, allocate, or provision R2 Data Catalog resources.
  • @gscdump/lakehouse/unsafe-raw: explicit escape hatch for package adapters and raw-metadata diagnostics. Application maintenance belongs on the stable maintenance subpath.
import { connectIcebergCatalog, listIcebergTables } from '@gscdump/lakehouse'

// Supply your catalog connection configuration.
const connection = await connectIcebergCatalog(config)
const tables = await listIcebergTables(connection)
console.log(tables)

For retry decisions, import isCommitRateLimited, isCommitServerError, or isCommitTransient from @gscdump/lakehouse/maintenance.

isCommitRateLimited matches HTTP 429. isCommitServerError matches transient R2 5xx responses. isCommitTransient matches either class and drives the package's append retry loop.

Use the dataset registry to define datasets. Raw Icebird table creation and append functions live under unsafe-raw for package adapters and diagnostics.

Opt-in manifest read limits

icebergManifests on @gscdump/lakehouse/unsafe-raw accepts an optional readLimits: ManifestReadLimits. Omission preserves the existing reader and caller-owned manifest cache behavior. No file resolver or host automatically enables these limits.

import { icebergManifests } from '@gscdump/lakehouse/unsafe-raw'

const manifests = await icebergManifests({
  metadata,
  resolver,
  readLimits: {
    maxObjectBytes: 2 * 1024 * 1024,
    maxTotalObjectBytes: 8 * 1024 * 1024,
    maxManifests: 8,
    maxEntries: 128,
    maxInflatedBlockBytes: 2 * 1024 * 1024,
    maxTotalInflatedBytes: 8 * 1024 * 1024,
    maxValueBytes: 1024 * 1024,
    maxDecodedValues: 100_000,
    maxHeaderBytes: 64 * 1024,
    maxSchemaDepth: 32,
  },
})

These are example admission values, not defaults or a total JavaScript heap guarantee. Every field is required and must be a positive safe integer; maxSchemaDepth must also be at most 128. The reader copies and validates the policy before its first asynchronous read. Unknown fields are rejected.

The per-object and cumulative source-byte checks happen after resolver.reader reports byteLength, before the reader requests slice(0, byteLength). Declared manifest lengths and returned slice lengths must match. An injected resolver can allocate or fetch inside reader itself; those operations are outside this boundary and need their own transport limits and immutable-object identity checks. No HEAD, catalog JSON, network-request, or freshness bound is implied by this option.

Manifest-list counts are admitted before record materialization and pruning; entry counts accumulate across all fetched manifests and blocks. V1 inline manifest counts are admitted before reads, with each file decoded once in a private per-call cache. Supplying manifestCache with readLimits is rejected: previously decoded cache entries carry no verifiable budget provenance. Defaults retain their existing cache behavior.

For Avro null and raw deflate blocks, decoded block bytes and their cumulative total are bounded. The existing pure-JavaScript inflater checks output growth before allocating or writing past that limit. Bounded DEFLATE returns an exact-size backing buffer so byte views do not retain spare inflater capacity; growth and final-copy buffers can temporarily overlap. Header framing is byte-bounded before JSON parsing; schema depth/work is checked before recursive compilation. Variable and fixed values are checked before views or UTF-8 decoding, and maxDecodedValues bounds decoding work across header/schema nodes and decoded value visits, including arrays of zero-byte nulls. Strict reads also reject invalid counts, varints, indexes, truncated values and inconsistent block sizes.

These limits do not measure JavaScript object/string overhead, the JSON parser's internal allocations, inflater tables, caller-retained output, injected resolver caches or total process memory. Output remains an eager array within the chosen count/byte/work policy. A refusal rejects the complete call; there is no eager fallback or partial success. Already-dispatched resolver reads cannot be physically cancelled by the AsyncBuffer interface, but a first failure stops new dispatches and prevents late responses from being decoded. Choose compatible limits and validate the complete consumer path before enabling them.

The same optional readLimits can be passed to resolveIcebergDataFiles, dataset.resolveDataFiles(..., { readLimits }), and the engine's listIcebergDataFiles. These wrappers capture and validate all ten values before catalog reads or lazy imports, then perform one bounded manifest walk. Per-object and aggregate limits remain independent upper bounds; a smaller aggregate limit does not expand the per-object allowance. Invalid, missing or unknown values are rejected even when the table has no current snapshot.

In this opt-in mode, decoded manifest, resolved-file and month caches are neither read nor populated: old cache entries cannot establish compliance with a new policy. The existing catalog configuration, snapshot pointer and table metadata caches remain in use, with their existing identity/freshness rules. Catalog JSON and those metadata caches are outside the Avro allocation bound. Existing partition, Site, search-type, date pruning and output ordering are unchanged.

This has an explicit warm-read cost. A default exact-range cache hit can avoid all Avro reads; a default month hit can need only the manifest list. A bounded call reads the list and every selected manifest again (subject to any injected resolver's own byte cache).

A connection from connectIcebergCatalog keeps the objects it reads in a per-connection byte cache. That cache holds at most 16 MiB of fetched bytes. If a read takes it over 16 MiB, it evicts the least recently read object first. It does not keep an object larger than 16 MiB, and it does not keep a failed read. Concurrent reads of one path share one S3 request. A successful write or delete of a path evicts that path. A bounded call reads through the uncached S3 resolver instead, so the connection keeps none of its manifest bytes after the call. It also does not reuse bytes that earlier unbounded calls cached. If you replace connection.resolver, the bounded call uses your resolver, and your resolver owns its retention. Each concurrent bounded call owns its own limits; the option does not impose a connection-wide or process-wide budget. There is no automatic fallback to cached or unbounded results after refusal. Validate retained files and request costs before enabling a policy in a host; this does not bound a remote SQL provider's internal manifest planner.

Append retries

Supply a durable commitRetry.appendId when a dataset append belongs to a job or receipt. Reuse that ID when the job retries, including retries after package upgrades.

Without an explicit ID, row appends derive a versioned ID from framed, typed record content. Row order and object key order do not change the ID.

If only a legacy content ID matches, the append throws LegacyAppendIdentityUnverifiable. The error includes legacyAppendId. The legacy ID cannot prove whether the committed rows match the retry. Verify the committed rows before supplying that legacy ID as an explicit appendId. If the rows differ, use a new durable ID for the new append. Do not retry with the error's legacy ID without checking the rows.

Before upgrading producers that derive IDs, drain their existing write tasks. Do not run old and new derived-ID writers concurrently against the same table. Older writers cannot recognize the new content ID format.

Batch appends already require an explicit appendId and retain that behavior.

Manifest merging

Use mergeIcebergManifests(connection, table, options) from @gscdump/lakehouse/maintenance to reduce current metadata reads for append-only v2 tables. The connection supplies the exact catalog and namespace. The operation preserves DATA files, their metadata and sequence numbers, and historical append snapshots and manifests. It groups entries by partition specification and writes encoded manifests up to four MiB. It bounds aggregate input and staged output to eight MiB each, with ten thousand live entries. Options can lower these limits.

Merged reports the snapshot ID, manifest counts, and file count. Unchanged avoids writes when merging cannot reduce the manifest count. Conflict leaves a concurrent UUID or main-snapshot change intact. Reload the current head before a later attempt. Err reports unsupported formats, deletes, invalid inventory, exceeded limits, or changed external authority. Infrastructure errors propagate. Non-null DATA metadata fields unsupported by the v2 writer, including encryption metadata, produce invalid-inventory.

Supply confirmAuthority to recheck external authority immediately before the catalog commit. Returning false prevents that commit; callback exceptions propagate. This callback does not create an atomic transaction with an external database. Supply cache to invalidate the captured namespace and table's cached snapshot pointer after success.

Failed or uncertain commits retain staged metadata. An uncertain response may follow a successful commit, so deleting staged files could break the table. The operation never expires historical snapshots or deletes historical manifests. Keep those objects while authenticated receipts or append retries still need them.

Table restoration

Use restoreIcebergTable(connection, table, options) from @gscdump/lakehouse/maintenance for an empty Catalog target. Restore every referenced metadata, manifest, and data object before registering the table. Preserve their S3 locations, table UUID, snapshot IDs, and immutable bytes. The caller owns the target and excludes concurrent writers through settlement.

Supply the saved metadata location, SHA-256, table UUID, head snapshot ID, and complete snapshot ID inventory. The operation checks the target, saved metadata bytes, and identity before registration. Metadata can use plain JSON or gzip. The byte limit applies to stored and expanded bytes. The SHA-256 always authenticates the stored object bytes before decompression. It registers with overwrite: false and verifies the returned identity. Existing tables produce target-exists, including tables with matching saved identities. Malformed bytes or mismatched identities produce backup-invalid before registration.

Supply confirmAuthority to recheck ownership before access, immediately before registration, and after acknowledgement. This callback does not create an atomic transaction with external state. Restored confirms the acknowledged table identity. It does not verify every referenced data object. After restoration, run the application's receipt, object hash, and query checks through fresh read sessions. Supply cache to invalidate its saved snapshot pointer after success.

Unknown means registration may have committed. Keep the restored objects and inspect fresh Catalog metadata before deciding the next action. The operation never retries, overwrites, or removes objects after an uncertain response. Other infrastructure failures propagate.

To detach a registration while retaining its objects, use dropIcebergTables(connection, tables, { purge: false }). The default still requests purging. Use explicit table names for a restore drill.

License

MIT

版本与发布记录

@gscdump/lakehouse 共发布 139 个版本,首次发布于 2026/07/03;过去 12 个月发布了 139 个版本。最新版本附带 npm provenance 构建来源证明,可追溯到公开的 CI 构建记录。

版本发布日期距今
7.15.0latest2026/11/16今天
7.14.02026/11/16今天
7.13.22026/11/151 天前
7.13.12026/11/152 天前
7.13.02026/11/124 天前
7.12.12026/11/116 天前

关于 @gscdump/lakehouse 的常见问题

@gscdump/lakehouse 是什么?

Dataset-agnostic Iceberg lakehouse layer + dataset registry for R2 Data Catalog producers (ADR-0021).

如何安装 @gscdump/lakehouse?

在项目目录运行 npm install @gscdump/lakehouse,或使用 pnpm add / yarn add / bun add。

@gscdump/lakehouse 支持 TypeScript 吗?

支持。包内自带 TypeScript 类型声明(package.json 声明了 types 入口),安装后即可获得类型提示,无需再安装 @types 包。

@gscdump/lakehouse 支持 ESM 还是 CommonJS?

以 ES Module 形式发布,推荐使用 import 引入。在 CommonJS 项目中,Node.js 20.19+ / 22.12+ 可以直接 require() 不含顶层 await 的 ES Module,更早的版本需要改用动态 import()。

@gscdump/lakehouse 需要什么版本的 Node.js?

package.json 的 engines 字段要求 Node.js >=22.13.0。版本不符时 npm 默认只给出警告,开启 engine-strict 后会拒绝安装。

@gscdump/lakehouse 可以商用吗?

@gscdump/lakehouse 使用 MIT 许可。宽松许可:允许商业使用、修改和再分发,通常只需在分发时保留原版权与许可声明。以上为概括,不构成法律意见,请以许可原文为准。

@gscdump/lakehouse 的下载量有多少?

根据 npm 官方下载统计,@gscdump/lakehouse 在 2026/10/17 – 2026/11/15 这 30 天内被下载 10,232 次,日均约 341 次。下载量会把 CI 构建、镜像同步和作为其他包依赖的安装一并计入,适合横向比较热度,不能直接等同于使用人数。

@gscdump/lakehouse 还在维护吗?

最新版本 7.15.0 发布于 2026/11/16(今天);至今共发布 139 个版本,其中过去 12 个月发布了 139 个;首次发布于 2026/07/03。发布频率只是参考信号:功能稳定的工具可能很少需要更新,建议结合源码仓库的 issue 和提交记录判断。