@dreamlake/dreamdb

This 0.5.10 build includes read-only SqlSession.open(refName, backend), asynchronous query and synchronous explain. See the SQL adapter guide. The browser SQL artifact loads on first open; Node includes it directly. This addition is not a claim that previously published npm versions support SQL.

Package scope: npm 0.5.10 updates the shared core. The one JavaScript API addition since 0.5.8 is Space.readEmbeddingColumn, first published in 0.5.9: a bounded, exact read of a whole Embedding column that refuses rather than returning a partial column. The rest is shared-core write and read cost reductions and fixes. Paged array indexes from 0.5.8, SQL from 0.5.6, and per-publication SDK/build diagnostics and the deferred compaction commit fix from 0.5.5 remain. Existing entity keys, geometry, structured arrays and query-local rerank controls remain. Rust operator APIs are not automatically JS exports. See the release notes, feature guide and SDK reference.

The DreamDB SDK — read and write, in the browser and on the server, compiled to WebAssembly from the same Rust core the CLI and Python SDK use.

There is no separate JavaScript implementation of the protocol, and that is the point. A hand-written port has to reproduce BLAKE3, canonical CBOR, spatial-key encoding, bucket headers and index layout bit for bit, forever. The previous TypeScript port did not, and its tests were all green anyway, because they were its own reader reading its own writer.

npm install @dreamlake/dreamdb

Reading

import { Space } from '@dreamlake/dreamdb'

const space = await Space.fromUri('https://bucket.s3.amazonaws.com/refs/my-dataset', null)
const hits = await space.queryVector('visual', queryVec, 24, 8)

Passing null for the backend uses direct fetch against the URI's base — enough for a public bucket. For anything else, supply a Backend.

Writing

import { Writer, PresignedBackend } from '@dreamlake/dreamdb'

const backend = new PresignedBackend({
  readBase: 'https://bucket.s3.amazonaws.com',
  mintPut: async (paths) => (await postJson('/api/dreamdb/sign', { paths })).urls,
  commitRef: async (path, opts, bytes) =>
    await postJson('/api/dreamdb/ref', {
      path,
      bytesBase64: base64(bytes),
      ifMatch: opts.ifMatch,
      ifNoneMatchStar: opts.ifNoneMatchStar,
    }),
})

const w = await Writer.open('https://bucket.s3.amazonaws.com/refs/my-dataset', backend)
await w.appendMany([
  {
    anchor: 1735689600000000000n,          // nanoseconds — pass a bigint
    visual: { kind: 'embedding', algorithm: 'dreamdb.lsh-cosine', vector: vec },
    caption: { kind: 'categorical', value: 'a red car' },
  },
])
const manifest = await w.commit()

Credentials never enter the browser: your server mints short-lived presigned PUT URLs, and performs the ref's compare-and-swap itself.

Anchors are nanoseconds, and you should pass a bigint

A number is accepted only while it is below Number.MAX_SAFE_INTEGER; above that the SDK throws rather than round it. This is not pedantry. The previous TypeScript SDK wrote anchors in microseconds specifically to stay under 2^53, and every record it produced reads as 1970 in any conformant reader.

Server-side authoring

Authoring adds dataset creation, layers, merge, compaction and history. It is Node only — the browser build exports a stub whose methods throw with an explanation.

import { Authoring } from '@dreamlake/dreamdb'   // resolves to the Node build

const a = await Authoring.create(
  'my-dataset',
  [
    { name: 'visual', kind: 'embedding', dim: 768, algorithm: 'dreamdb.lsh-cosine' },
    { name: 'caption', kind: 'scalar', valueType: 'categorical', required: false },
  ],
  'https://bucket.s3.amazonaws.com',
  backend,
)
await a.writer().appendMany(samples)
await a.compact(null, 1, 0)

The split is a size boundary with a mechanism behind it. create and the embedding-layer builders construct a SpatialDispatcher, which is an enum — so one reachable construction makes every index family's build code reachable (IVF, IMI, LSH, Vamana, AdaIVF, plus codebook training). Measured: 484 KB gzipped for the browser build versus 607 KB with authoring included.

Embedding fields declaring dreamdb.ivf-cosine or dreamdb.imi-cosine cannot be created here at all: those index families need training data that does not exist at create time. Create with the default dreamdb.lsh-cosine and attach a trained index as a layer, or build it with the CLI.

Entry points

<script type="module">
  import ready, { Space } from 'https://esm.sh/@dreamlake/dreamdb/web'
  await ready()                 // fetches and instantiates the wasm
</script>

Implementing a Backend

Only get is required; a get-only backend is the read-only v1 shape and still works for reading. Implement getStream to keep full-object reads bounded when DreamDB must scan a historical oversized inline Track. Write entry points check for put up front and refuse with a message naming what is missing.

interface Backend {
  // `range` is half-open: start is included, end is excluded.
  get(path: string, range?: { start: number; end: number })
    : Promise<Uint8Array | { bytes: Uint8Array; etag?: string; totalLength?: number }>
  getStream?(path: string)
    : Promise<ReadableStream<Uint8Array> | {
        stream: ReadableStream<Uint8Array>
        etag?: string
        totalLength?: number
      }>
  head?(path: string): Promise<{ exists?: boolean; etag?: string; size?: number }>
  put?(path: string, bytes: Uint8Array,
       opts?: { ifMatch?: string; ifNoneMatchStar?: boolean })
    : Promise<{ status: 'created' | 'exists' | 'casFailed'; etag?: string }>
  delete?(path: string): Promise<void>
  list?(prefix: string): Promise<string[]>
}

getStream chunks must themselves be bounded; returning one object-sized Uint8Array is functionally valid but does not provide the bounded-memory profile. The SDK hashes every chunk and exposes no decoded result until the complete response matches its content address. Existing backends without getStream continue through get and are intentionally classified as the unbounded compatibility profile.

Two things that will bite you

Your bucket's CORS rule must include ExposeHeaders: ["ETag"].

A ref advances by compare-and-swap. The SDK sends If-Match: <etag> when it knows the ref's current version and If-None-Match: * when it does not. Without ExposeHeaders, the browser can read the response body but not the ETag header, so the SDK never learns the current version, falls back to create-only, and the second commit to any ref fails while the first succeeded. That asymmetry is what makes it hard to recognise.

{
  "AllowedOrigins": ["https://your.app"],
  "AllowedMethods": ["GET", "HEAD", "PUT"],
  "AllowedHeaders": ["*"],
  "ExposeHeaders": ["ETag", "Content-Range", "Content-Length"]
}

Any HTTP layer you put in front of the bucket must honour Range.

DreamDB reads exact vectors by byte offset. A server that ignores Range and returns 200 with the whole object makes every ranged read return offset 0, and search then returns plausible-looking results with cosine scores of 0.0000 — indistinguishable from a genuinely corrupt index. (python3 -m http.server does exactly this.)

Conformance

dreamdb-conformance/vectors/ holds 85 language-agnostic JSON vectors. This package runs the 48 that map to pure functions on its surface — canonical CBOR, BLAKE3 multihash, spatial keys, time anchors and buckets, address round-trips, modality parsing, HTTP range translation — and reports the other 37 by name and reason rather than omitting them silently.

node test/conformance.mjs

The first run of that suite found a real bug in this package's HTTP range translation, which is roughly the point.

Licence

MIT OR Apache-2.0

Publisher identity

Writer/Authoring publications record @dreamlake/dreamdb and the actual npm version in the existing Manifest writer tag. Use writer.setApplication(name, revision) (or authoring.writer().setApplication) for optional application metadata on later publications, and Space.history() to inspect it. Application components are 1–128 UTF-8 bytes without control characters. Existing payloads are not attributed to their latest publisher; provenance is diagnostic, not an execution attestation or compatibility check.