@harperfast/hnsw

中文文档
0.4.0发布于 2026/09/22Apache-2.0TypeScriptCommonJS零依赖近 30 天下载 10万

Persistent, incrementally-maintained, concurrently-searchable native HNSW for Node.js: a memory-mapped fixed-slot graph file with off-event-loop search, seqlock concurrency, int8 asymmetric distance, and bitset/predicate filtering.

原始使用文档
文档目录6 节
  1. Install
  2. Usage
  3. Storage precision
  4. Benchmarks
  5. Status
  6. License

@harperfast/hnsw

Persistent, incrementally-maintained, concurrently-searchable HNSW vector index for Node.js — a native (Rust) traversal engine over a memory-mapped fixed-slot graph file.

Most HNSW libraries for Node either keep the graph in JS objects (slow per-visit cost, GC pressure) or wrap an in-memory C++ index with no durable incremental persistence. This one is built around a different contract:

  • The file is the index. One memory-mapped file per index: fixed-size node slots (quantized vector + neighbor ids, page-grouped so slots never straddle page boundaries), an in-file upper-layer region, an id freelist, and a durability watermark. Reopen is instant — no rebuild, no sidecars.
  • Search never touches the JS event loop. Queries run on the libuv thread pool with one N-API crossing each; traversal is zero-copy against the mapping with SIMD (AVX2) int8 asymmetric-cosine distance.
  • Reads and writes are genuinely concurrent. Per-slot seqlocks, no global locks on the search path. Measured on one Linux box at 1M × 768-d (ef 512): 6,300+ QPS aggregate across 8 search threads while a writer sustains ~1,100 inserts/s, p50 ≈ 1 ms.
  • Index build scales with cores. insertBatch takes a whole chunk across one N-API crossing and fans the inserts out over worker threads inside the native module, off the event loop — the per-slot seqlocks are what make concurrent inserts safe. Scaling curve in DESIGN.md §11.
  • Incremental by design. Insert, update in place, delete with neighbor repair; deleted ids are reused via the freelist, so churn never inflates the graph. Reverse-edge overflow uses coverage-aware pruning (a bounded RobustPrune) — measured recall@10 of 0.999 at 1M (768-d int8, ef 512) on a calibrated Gaussian-mixture corpus.
  • Filtering built in. Allow-bitset filtering (zero callbacks), or a JS predicate evaluated in pipelined batches over a threadsafe function while traversal keeps expanding — a busy event loop costs speculative overshoot, never search-thread stalls.
  • Two integration modes. Standalone (the library allocates ids and maintains the graph: insert/remove/search), or mirroring (writeNodeRaw/clearNode: a host application that already maintains an HNSW graph mirrors it in and gets the native search path — this is how Harper integrates it).

Durability is deliberately relaxed: the file is msync'd on a cadence with a transaction watermark, and the intended recovery model is "replay indexing from the watermark" against the host's authoritative record store. Approximate indexes don't need per-commit fsyncs; they need cheap, bounded catch-up. See DESIGN.md for the format, the concurrency model, measured baselines, and the reasoning behind every trade.

Install

npm install @harperfast/hnsw

Prebuilt bindings ship as platform-specific optionalDependencies (@harperfast/hnsw-<platform>-<arch>[-glibc]) for linux-x64, linux-arm64, darwin-arm64, and win32-x64. Platforms without a published binding (musl, darwin-x64, win32-arm64) build from source on install when a Rust toolchain is present, and throw a clear error otherwise. Linux x86_64 is the performance target (AVX2 + kernel-lock crash recovery); macOS and Windows are functional (no lock takeover — bounded degradation instead).

Usage

const { Plane } = require('@harperfast/hnsw');

// keyCap 40 (min 8): each slot carries up to 40 bytes of the host's key inline (longer keys overflow)
// layer-0 cap: 64 (= 4M for M 16) in general — for these 768-d vectors it stays within 0.35 pt of
// cap 128's recall@10 with a 19% smaller slot; 32 for narrow vectors on a plane that outgrows RAM,
// the smallest slot, but 1.3–2.2 pts below cap 128 under ef 1024 at 4M × 128-d (DESIGN.md §10)
const plane = Plane.create('/data/vectors.hnsw', 768, 64, 10_000_000, 40);
// ... or with the finer storage precision (see below):
// const plane = Plane.create('/data/vectors.hnsw', 128, 32, 10_000_000, 40, undefined, 'int16');
const id = plane.insert(myFloat32Vector, Buffer.from(myRecordKey));
// bulk load: one crossing per chunk, inserted in parallel off the event loop; `ids` is in input
// order, keys are concatenated with SearchHits-style ends. Records the plane cannot hold come
// back in `rejected` (index + code) with 0xFFFFFFFF in their slot; a full plane rejects the
// promise with an error that still carries `ids`.
const { ids, rejected } = await plane.insertBatch(chunkVectors /* count × dims */, chunkKeys, chunkKeyEnds, 8 /* threads */);
// parallel typed arrays, ascending by distance; hit i's key is keys.subarray(keyEnds[i-1] ?? 0, keyEnds[i])
const { ids, distances, keys, keyEnds } = await plane.search(queryVector, 10, 512);

// filtered: allow-bitset over node ids
const allowed = new Uint8Array(Math.ceil(plane.idHighWater() / 8));
// ... set bits ...
const filtered = await plane.search(queryVector, 10, 512, allowed);

// or a JS predicate, batched off the event loop
const predicated = await plane.searchWithPredicate(queryVector, 10, 512, (ids) =>
    Uint8Array.from(ids, (id) => (isVisible(id) ? 1 : 0))
);

Storage precision

Vectors are stored quantized with a per-vector symmetric scale. The default, 'int8', maps each component to max|c|/127 — about 0.8% of the vector's largest component per element, which is usually close enough to rank candidates but not to score them, so callers typically rerank the returned hits against exact vectors. 'int16' maps to max|c|/32767 instead: ~256× finer, about 0.003% per element. Whether that is close enough to drop the rerank is a question about your corpus and your accuracy budget, and this package cannot answer it for you — validate it against your own data before turning a rerank off. It costs one more byte per dimension per slot — at cap 32, +40% at 128 dims (320 → 448 B), +89% at 1536 (1728 → 3264 B) — so it suits small-to-mid dimensionality, while int8 stays the right choice for wide embeddings, where doubling the bytes each traversal scans pushes search into the memory-bound regime. The choice is fixed at create and cannot be changed without a rebuild. Int16 planes carry a newer format version, so an older build of this package refuses to open one rather than misreading its slot layout; plane.precision reports an open plane's codec.

A plane is derived state; when the host must stop maintaining one and cannot delete the file (Windows sharing violations while another process maps it), invalidatePlane(path) — or plane.invalidateFile() through a handle the host already holds — durably marks it unadoptable: a one-way in-band latch (watermark reads 0, Plane.open refuses) plus a fsync'd <path>.stale sidecar (stalePathFor(path), which open also refuses). It throws only when neither marker lands. Hosts delete both files and rebuild.

Full API in index.d.ts.

Benchmarks

cargo run --release --bin bench -- 1000000 768 100 512 /tmp/bench.hnsw 64 8 builds a 1M × 768-d graph on a calibrated Gaussian-mixture corpus, reports p50/p95/p99, per-visit cost, brute-force recall@10, and a concurrent-throughput pass. A ninth argument selects the storage precision (int8, int16, or both to build and measure one plane of each), and every run prints a kernel microbenchmark first. Numbers from the design work (Linux, single box): p50 0.75 ms, 0.33 µs/visit, recall@10 0.999 — ~9× the wall-clock and ~13× the per-visit cost of a well-optimized pure-JS implementation of the same graph at equal recall.

Status

Extracted from the Harper vector-index engine; the format (v8 for int8 planes, v9 for int16) and API are young and may change with a version bump + reindex (an older format version fails to open; rebuild). Roadmap: prebuilds, binary-quantized slot format (~4× smaller traversal plane), Matryoshka dimension truncation, mremap growth, index slicing with native top-k merge.

License

Apache-2.0

版本与发布记录

@harperfast/hnsw 共发布 4 个版本,首次发布于 2026/09/01;过去 12 个月发布了 4 个版本。

版本发布日期距今
0.4.0latest2026/09/221 个月前
0.3.02026/09/171 个月前
0.2.12026/09/042 个月前
0.1.02026/09/012 个月前

关于 @harperfast/hnsw 的常见问题

@harperfast/hnsw 是什么?

Persistent, incrementally-maintained, concurrently-searchable native HNSW for Node.js: a memory-mapped fixed-slot graph file with off-event-loop search, seqlock concurrency, int8 asymmetric distance, and bitset/predicate filtering.

如何安装 @harperfast/hnsw?

在项目目录运行 npm install @harperfast/hnsw,或使用 pnpm add / yarn add / bun add。

@harperfast/hnsw 支持 TypeScript 吗?

支持。包内自带 TypeScript 类型声明(package.json 声明了 types 入口),安装后即可获得类型提示,无需再安装 @types 包。

@harperfast/hnsw 支持 ESM 还是 CommonJS?

以 CommonJS 形式发布,可以直接 require();在 ES Module 项目中也可以用 import 引入(Node.js 会把 module.exports 作为默认导出)。

@harperfast/hnsw 需要什么版本的 Node.js?

package.json 的 engines 字段要求 Node.js >=18。版本不符时 npm 默认只给出警告,开启 engine-strict 后会拒绝安装。

@harperfast/hnsw 可以商用吗?

@harperfast/hnsw 使用 Apache-2.0 许可。宽松许可:允许商业使用、修改和再分发,并附带明确的专利授权;分发时需保留许可与 NOTICE 文件,修改过的文件要注明改动。以上为概括,不构成法律意见,请以许可原文为准。

@harperfast/hnsw 的下载量有多少?

根据 npm 官方下载统计,@harperfast/hnsw 在 2026/10/18 – 2026/11/16 这 30 天内被下载 99,721 次,日均约 3,324 次。下载量会把 CI 构建、镜像同步和作为其他包依赖的安装一并计入,适合横向比较热度,不能直接等同于使用人数。

@harperfast/hnsw 还在维护吗?

最新版本 0.4.0 发布于 2026/09/22(1 个月前);至今共发布 4 个版本,其中过去 12 个月发布了 4 个;首次发布于 2026/09/01。发布频率只是参考信号:功能稳定的工具可能很少需要更新,建议结合源码仓库的 issue 和提交记录判断。