@gscdump/lakehouse
Dataset-agnostic Iceberg catalog, schema, and dataset-registry primitives for R2 Data Catalog producers.
Install
npm install @gscdump/lakehouse
Node.js 22 or newer is required.
Public surfaces
@gscdump/lakehouse: connect to a catalog, define datasets, derive table specifications, resolve data files, and use bigint-safe serialization.@gscdump/lakehouse/bigint: bigint-safe JSON helpers without catalog or Node dependencies.@gscdump/lakehouse/schema: lightweight shared schema constants.@gscdump/lakehouse/maintenance: stable commit classification, manifest merging, and orphan cleanup workflows.@gscdump/lakehouse/provisioning: adopt, allocate, or provision R2 Data Catalog resources.@gscdump/lakehouse/unsafe-raw: explicit escape hatch for package adapters and raw-metadata diagnostics. Application maintenance belongs on the stable maintenance subpath.
import { connectIcebergCatalog, listIcebergTables } from '@gscdump/lakehouse'
// Supply your catalog connection configuration.
const connection = await connectIcebergCatalog(config)
const tables = await listIcebergTables(connection)
console.log(tables)
For retry decisions, import isCommitRateLimited, isCommitServerError, or
isCommitTransient from @gscdump/lakehouse/maintenance.
isCommitRateLimited matches HTTP 429.
isCommitServerError matches transient R2 5xx responses.
isCommitTransient matches either class and drives the package's append retry loop.
Use the dataset registry to define datasets.
Raw Icebird table creation and append functions live under unsafe-raw for package adapters and diagnostics.
Opt-in manifest read limits
icebergManifests on @gscdump/lakehouse/unsafe-raw accepts an optional
readLimits: ManifestReadLimits. Omission preserves the existing reader and
caller-owned manifest cache behavior. No file resolver or host automatically
enables these limits.
import { icebergManifests } from '@gscdump/lakehouse/unsafe-raw'
const manifests = await icebergManifests({
metadata,
resolver,
readLimits: {
maxObjectBytes: 2 * 1024 * 1024,
maxTotalObjectBytes: 8 * 1024 * 1024,
maxManifests: 8,
maxEntries: 128,
maxInflatedBlockBytes: 2 * 1024 * 1024,
maxTotalInflatedBytes: 8 * 1024 * 1024,
maxValueBytes: 1024 * 1024,
maxDecodedValues: 100_000,
maxHeaderBytes: 64 * 1024,
maxSchemaDepth: 32,
},
})
These are example admission values, not defaults or a total JavaScript heap
guarantee. Every field is required and must be a positive safe integer;
maxSchemaDepth must also be at most 128. The reader copies and validates the
policy before its first asynchronous read. Unknown fields are rejected.
The per-object and cumulative source-byte checks happen after resolver.reader
reports byteLength, before the reader requests slice(0, byteLength).
Declared manifest lengths and returned slice lengths must match. An injected
resolver can allocate or fetch inside reader itself; those operations are
outside this boundary and need their own transport limits and immutable-object
identity checks. No HEAD, catalog JSON, network-request, or freshness bound is
implied by this option.
Manifest-list counts are admitted before record materialization and pruning;
entry counts accumulate across all fetched manifests and blocks. V1 inline
manifest counts are admitted before reads, with each file decoded once in a
private per-call cache. Supplying manifestCache with readLimits is rejected:
previously decoded cache entries carry no verifiable budget provenance. Defaults
retain their existing cache behavior.
For Avro null and raw deflate blocks, decoded block bytes and their cumulative
total are bounded. The existing pure-JavaScript inflater checks output growth
before allocating or writing past that limit. Bounded DEFLATE returns an exact-size
backing buffer so byte views do not retain spare inflater capacity; growth and
final-copy buffers can temporarily overlap. Header framing is byte-bounded
before JSON parsing; schema depth/work is checked before recursive compilation.
Variable and fixed values are checked before views or UTF-8 decoding, and
maxDecodedValues bounds decoding work across header/schema nodes and decoded
value visits, including arrays of zero-byte nulls. Strict reads also reject
invalid counts, varints, indexes, truncated values and inconsistent block sizes.
These limits do not measure JavaScript object/string overhead, the JSON parser's internal allocations, inflater tables, caller-retained output, injected resolver caches or total process memory. Output remains an eager array within the chosen count/byte/work policy. A refusal rejects the complete call; there is no eager fallback or partial success. Already-dispatched resolver reads cannot be physically cancelled by the AsyncBuffer interface, but a first failure stops new dispatches and prevents late responses from being decoded. Choose compatible limits and validate the complete consumer path before enabling them.
The same optional readLimits can be passed to resolveIcebergDataFiles,
dataset.resolveDataFiles(..., { readLimits }), and the engine's
listIcebergDataFiles. These wrappers capture and validate all ten values before
catalog reads or lazy imports, then perform one bounded manifest walk. Per-object
and aggregate limits remain independent upper bounds; a smaller aggregate limit
does not expand the per-object allowance. Invalid, missing or unknown values are
rejected even when the table has no current snapshot.
In this opt-in mode, decoded manifest, resolved-file and month caches are neither read nor populated: old cache entries cannot establish compliance with a new policy. The existing catalog configuration, snapshot pointer and table metadata caches remain in use, with their existing identity/freshness rules. Catalog JSON and those metadata caches are outside the Avro allocation bound. Existing partition, Site, search-type, date pruning and output ordering are unchanged.
This has an explicit warm-read cost. A default exact-range cache hit can avoid all Avro reads; a default month hit can need only the manifest list. A bounded call reads the list and every selected manifest again (subject to any injected resolver's own byte cache).
A connection from connectIcebergCatalog keeps the objects it reads in a
per-connection byte cache. That cache holds at most 16 MiB of fetched bytes.
If a read takes it over 16 MiB, it evicts the least recently read object
first. It does not keep an object larger than 16 MiB, and it does not keep a
failed read. Concurrent reads of one path share one S3 request. A successful
write or delete of a path evicts that path. A bounded call reads through the uncached
S3 resolver instead, so the connection keeps none of its manifest bytes after
the call. It also does not reuse bytes that earlier unbounded calls cached. If
you replace connection.resolver, the bounded call uses your resolver, and your
resolver owns its retention. Each concurrent bounded call owns its own limits;
the option does not impose a connection-wide or process-wide budget. There is
no automatic fallback to cached or unbounded results after refusal. Validate
retained files and request costs before enabling a policy in a host; this does
not bound a remote SQL provider's internal manifest planner.
Append retries
Supply a durable commitRetry.appendId when a dataset append belongs to a job or receipt.
Reuse that ID when the job retries, including retries after package upgrades.
Without an explicit ID, row appends derive a versioned ID from framed, typed record content. Row order and object key order do not change the ID.
If only a legacy content ID matches, the append throws LegacyAppendIdentityUnverifiable.
The error includes legacyAppendId.
The legacy ID cannot prove whether the committed rows match the retry.
Verify the committed rows before supplying that legacy ID as an explicit appendId.
If the rows differ, use a new durable ID for the new append.
Do not retry with the error's legacy ID without checking the rows.
Before upgrading producers that derive IDs, drain their existing write tasks. Do not run old and new derived-ID writers concurrently against the same table. Older writers cannot recognize the new content ID format.
Batch appends already require an explicit appendId and retain that behavior.
Manifest merging
Use mergeIcebergManifests(connection, table, options) from @gscdump/lakehouse/maintenance
to reduce current metadata reads for append-only v2 tables.
The connection supplies the exact catalog and namespace.
The operation preserves DATA files, their metadata and sequence numbers, and historical append snapshots and manifests.
It groups entries by partition specification and writes encoded manifests up to four MiB.
It bounds aggregate input and staged output to eight MiB each, with ten thousand live entries.
Options can lower these limits.
Merged reports the snapshot ID, manifest counts, and file count.
Unchanged avoids writes when merging cannot reduce the manifest count.
Conflict leaves a concurrent UUID or main-snapshot change intact.
Reload the current head before a later attempt.
Err reports unsupported formats, deletes, invalid inventory, exceeded limits, or changed external authority.
Infrastructure errors propagate.
Non-null DATA metadata fields unsupported by the v2 writer, including encryption metadata, produce invalid-inventory.
Supply confirmAuthority to recheck external authority immediately before the catalog commit.
Returning false prevents that commit; callback exceptions propagate.
This callback does not create an atomic transaction with an external database.
Supply cache to invalidate the captured namespace and table's cached snapshot pointer after success.
Failed or uncertain commits retain staged metadata. An uncertain response may follow a successful commit, so deleting staged files could break the table. The operation never expires historical snapshots or deletes historical manifests. Keep those objects while authenticated receipts or append retries still need them.
Table restoration
Use restoreIcebergTable(connection, table, options) from @gscdump/lakehouse/maintenance for an empty Catalog target.
Restore every referenced metadata, manifest, and data object before registering the table.
Preserve their S3 locations, table UUID, snapshot IDs, and immutable bytes.
The caller owns the target and excludes concurrent writers through settlement.
Supply the saved metadata location, SHA-256, table UUID, head snapshot ID, and complete snapshot ID inventory.
The operation checks the target, saved metadata bytes, and identity before registration.
Metadata can use plain JSON or gzip. The byte limit applies to stored and expanded bytes.
The SHA-256 always authenticates the stored object bytes before decompression.
It registers with overwrite: false and verifies the returned identity.
Existing tables produce target-exists, including tables with matching saved identities.
Malformed bytes or mismatched identities produce backup-invalid before registration.
Supply confirmAuthority to recheck ownership before access, immediately before registration, and after acknowledgement.
This callback does not create an atomic transaction with external state.
Restored confirms the acknowledged table identity. It does not verify every referenced data object.
After restoration, run the application's receipt, object hash, and query checks through fresh read sessions.
Supply cache to invalidate its saved snapshot pointer after success.
Unknown means registration may have committed.
Keep the restored objects and inspect fresh Catalog metadata before deciding the next action.
The operation never retries, overwrites, or removes objects after an uncertain response.
Other infrastructure failures propagate.
To detach a registration while retaining its objects, use dropIcebergTables(connection, tables, { purge: false }).
The default still requests purging. Use explicit table names for a restore drill.