@cloudbitmaps/gcs

Google Cloud Storage for CloudBitmaps. One bucket holds your generations and their pointers, so there is no second service to run.

ESM-only, Node ≥ 22.12. Use import; for require(), Jest and TypeScript, see CommonJS, Jest and TypeScript.

Pre-1.0: the API and the on-disk format can still change. On npm these docs match the version they ship in; on GitHub they describe main, and the changelog says what each release changed.

Install

pnpm add @cloudbitmaps/roaring @cloudbitmaps/gcs

Install the same version of every @cloudbitmaps/* package: they release together, and each pins the exact version of core it was built with.

@google-cloud/storage (^8.1.0) is a real dependency of this package, so installing it is the whole step. A client you pass must come from 8.1.0 or later: the driver's own tests pass on 8.1.0 and fail on every older release tried (8.0.1, 8.0.0, and 7.x from 7.0.0 to 7.22.0). On 8.0.1 the unit tests pass, but the integration test of a resumable upload's tail size against the emulator fails with a reset connection. On 8.0.0 and on 7.x the unit tests fail: a read the driver cuts off at readTimeoutMs leaves its connection open, a read whose first answer is cut off part-way is not retried to success (the call never settles), and a read the driver refuses does not release its request. @cloudbitmaps/core, the engine underneath, arrives with it and you never name it.

Use

import { CloudRoaring } from '@cloudbitmaps/roaring';
import { GcsStorage } from '@cloudbitmaps/gcs';

const store = new CloudRoaring({
  storage: new GcsStorage({ bucket: 'bitmaps', prefix: 'cr' }),
});

await store.load({ segment: 'active-users' }, [1, 2, 3]);
await store.segment('active-users').has(2); // true

It builds its own client from Application Default Credentials. Any other key is refused by name.

Before production

• This backend is outside the validated envelope. It passes the same conformance suites as S3, but the proven scale and the calibration were done on S3. You are an early user.
• Run await store.preflight() once before the first load. It writes, tries to write again, reads, lists and deletes one probe object, and reads the registry and the bucket's lifecycle rules (storage.buckets.get for the last), and answers { ok, checks }. It does not check a read after a delete or versioning, among others: the full list. A first run is eight requests here.
• Grant the bucket's object permissions. The library creates, reads, lists and deletes objects: roles/storage.objectAdmin covers all four.
• The write-once guarantee rides GCS preconditions (ifGenerationMatch), so nothing else needs enabling.
• Never add a lifecycle rule that expires current objects or the registry/ prefix.
• Check the bucket's lifecycle rules with await backend.checkLifecycle(). It reads the bucket's metadata with one request (bounded by readTimeoutMs, when set) and answers problem for a Delete rule that applies to live objects and can match your prefix, ok, or unchecked. It needs storage.buckets.get, which a load never uses, store.preflight()'s lifecycle step does, and roles/storage.objectAdmin does not include: without it the answer is unchecked, naming the permission (check the lifecycle rules from code).
• The SDK does not retry downloads; the driver does. In @google-cloud/storage 8.x (checked on 8.1.0) and 7.x (checked on 7.22.0, a release outside this driver's range), a download the SDK retries after any status it retries (408, 429, 500, 502, 503 or 504) can crash the process with ERR_STREAM_UNABLE_TO_PIPE, thrown outside any promise, even though the retried request succeeded. The client GcsStorage builds therefore sends each download once, and the driver runs a download again itself, up to three more times with backoff, after a connection fault (refused, reset, timed out, a DNS failure, a body cut off) or a 408, 429, 500, 502, 503 or 504, and after nothing else (not a missing credentials file or a TLS failure). That holds for every caller, including a store built with retry: false. The client's listings, deletes, resumable uploads and the metadata read that settles a write keep its retry settings (under which the SDK sends a delete with no precondition once); a single-request conditional write (a registry row, a small generation) is sent without them. A client you pass as client keeps its own retries for those requests: its downloads go through a twin of it, built from its own class with the same credentials object, endpoint, project, user agent, timeout, checksum generator and interceptors and the SDK's retries off. A Storage client is always used through its twin, which must have taken the settings that decide where and how a download goes: the same credentials object, URL and endpoint, and retry settings of its own, which are then turned off over any its class set. The backend refuses, when it is built, a client whose twin would not (a subclass that builds from options of its own), one whose class overrides bucket (as a test double built on Storage does), and one whose bucket is replaced on the instance (as by a test stub). Stub Storage.prototype (or Bucket.prototype, File.prototype) instead, which reaches the twin; a stub on the client's instance does not. A test double that is not a Storage client is used as it is when it has retryOptions: { autoRetry: false }, and refused without it; with the SDK's module mocked, pass the mock as client. What is set on a client after the backend was built, other than an interceptor, is not carried to the twin.
• A client timeout does not bound a download on @google-cloud/storage 8.x, so a read whose server stalls waits for it. Measured against a local server that never answers: still pending after 12 s with timeout: 2000. readTimeoutMs does, and it is off unless you set it. It bounds each read as a whole: a generation's tail (with the metadata read it falls back on for an empty object), a range of it, or a registry row. One deadline covers every attempt the driver makes and the backoff between them, from the call into the driver (a credential fetch and any wait for a socket count) to the end of the body. It counts time the process spends busy too: Node runs a due timer before it reads a socket, so a synchronous stretch longer than the timeout fails the reads in flight even when their responses have arrived. When it passes, the read throws TransientError naming the read and the timeout, and no further attempt starts; the store's read retry runs it again, so with readTimeoutMs: 2_000 a read that stalls on every attempt fails after about 8.35 s (4 × 2 s plus up to 0.35 s of backoff; measured: 8.1 to 8.2 s). Uploads, deletes and listings are not timed.
• A read that times out before its server answers leaves its connection open. The SDK cannot cancel a request whose response has not begun, so that request holds its connection until the server answers or closes it, and until then keeps a Node process from exiting: one per driver read, up to four per call through the store's retry, so an outage at N reads a second adds about 4N held connections each second. On every version this driver's range admits (8.1.0 and later) a read cut off after its response began closes its own; on 8.0.0 and 7.x (checked on 7.22.0), both outside the range, it stays open too. A 404 whose error body arrives after the deadline is reported as a TransientError, not NotFoundError.
• Downloads go out on Node's global agent, so one the driver cuts off or refuses resets nothing else. When a download fails part-way, the SDK destroys the HTTP agent it went out on, and its own agent is one keep-alive pool the whole process shares, so every other request in flight would be reset with it, uploads included. The SDK never destroys the global agent. Measured on 8.1.0 with no timeout: a body cut off part-way reset a 64-byte upload and a registry write on a second backend on the SDK's agent, and on the global agent all of them completed. The cost: the global agent closes a connection idle for 5 seconds (the SDK's own pool does not), so a read after a longer pause opens a new connection, with its TCP and TLS handshake, and downloads share that agent with any other http or https request in the process. Raising https.globalAgent.options.timeout keeps idle connections longer.
• A transient failure of a write throws TransientError, and the write may or may not have landed. Re-run the call, or check store.generations(ref). The driver sends each registry write once; a load that gets no answer reads the row and sends a fresh compare-and-swap from it, at most three times. A generation's object of up to simpleUploadThresholdBytes is sent again, up to three more times, only after a 429 or 503; every object carries a random id in its metadata, so a first send that landed is told from another writer's object. A load whose row writes all go unanswered throws TransientError and keeps its object. The waits before an object's re-sends use real timers: the store's injected clock does not reach them, and GcsStorage takes no clock, so a test that throttles one object write three times waits about 3.5 s at most (the publish's waits before a fresh row write do use the store's clock).

The production checklist covers each item, and the ones every backend shares: backups of the data and the registry, and a schedule for your loads.

• The driver redacts the credentials it recognises from an error before it throws it. The SDK keeps the response it got on the error (response), and through it the request it sent, with the bearer token in its Authorization header, and the HTTP agent it shares with other calls; a resumable upload's or a token request's error keeps the request's config. On a download, and wherever a body lists no errors, the SDK makes the response body the error's message, so a proxy's page or a JSON body that quotes the request puts the token in the message. In the error the driver throws, the value under a credential's name is replaced by [redacted] (authorization, x-goog-encryption-key, x-goog-api-key, X-Goog-Signature, X-Goog-Credential, access_token, refresh_token, client_secret, private_key and the like; a service account's client_email and private_key_id are kept), and so are the value after one in a header list or under one in a map, the value beside one in a record (a { name, value } pair, a { th, td } row; a number, a boolean, and the message, reason and location of a Google error item are kept), and, anywhere in its text, a credential header line; an Authorization value, an encryption key, an API key, an OAuth token or client secret, or a signed URL's signature or credential after its name; a cookie header's value after its name; a bearer token after its scheme; a Google access token (ya29.) or a JWT on its own; a V2 signed URL's Signature=; a PEM private key's body; the text of an element named for a credential, such as the <CanonicalRequest> a service quotes back; and the text of the cell after one that names a credential (a <th>/<td> row, a <dt>/<dd> pair, after a heading or a caption too). A credential in another form is not recognised (a header table laid out by column, a user and password written into a URL, a colon written as an HTML entity such as : in a body a download quotes as it came), and a text these steps cannot finish is replaced whole. An SDK error that carries its transport or shows a credential, and every error it reaches (its cause chain, an AggregateError's errors), is thrown as a copy: a native error with the same prototype, name, message, stack, code, status and errors, with response reduced to its status and status text, so classification is unchanged, without config, the request, its agent or socket, any function or accessor property, or anything nested deeper than ten levels. The live error is never changed, and an error that holds only plain data and shows no credential is thrown as it is.

Documentation

• Getting started
• Before production
• API reference
• Changelog

License

Apache-2.0