@backloghq/termlog
Log-structured full-text search index — segment-based posting lists with LSM compaction, BM25 ranking, zero native dependencies.
Install
npm install @backloghq/termlog
Usage
import { TermLog } from "@backloghq/termlog";
const index = await TermLog.open({ dir: "./my-index" });
await index.add("doc-1", "the quick brown fox");
await index.add("doc-2", "the lazy dog");
await index.flush();
const results = await index.search("fox", { limit: 10 });
// [{ docId: "doc-1", score: 0.655... }] (BM25 — exact value depends on corpus)
await index.remove("doc-1");
await index.close();
Why
Existing FTS engines (Lucene, Tantivy) require native deps or a JVM. Most pure-JS alternatives serialize the index to a single in-memory blob — fine for small corpora, but they hit per-file size cliffs in the tens of thousands of documents. Termlog uses immutable on-disk segments with LSM compaction so the corpus scales without those ceilings.
Architecture
- Posting lists —
term → [docId, tf], compressed with VByte / delta encoding. (Positions reserved for a future release.) - Term dictionary — sorted on disk; binary search for lookup.
- Segments — self-contained immutable files (term dict + postings). New writes create a new segment. Compaction merges N segments into 1.
- Query execution — boolean (AND/OR) via posting iterators (zigzag merge for AND, union scan for OR), BM25 scoring on top.
- Storage — abstracted via
StorageBackend; local FS by default, S3 via @backloghq/termlog-s3.
S3 backend
S3 support is provided by the companion package @backloghq/termlog-s3:
npm install @backloghq/termlog @backloghq/termlog-s3
import { TermLog } from "@backloghq/termlog";
import { S3Backend } from "@backloghq/termlog-s3";
import { S3Client } from "@aws-sdk/client-s3";
const index = await TermLog.open({
dir: "my-index",
backend: new S3Backend({
client: new S3Client({ region: "us-east-1" }),
bucket: "my-bucket",
prefix: "my-index/",
}),
});
See the termlog-s3 README for IAM permissions, lifecycle rules, and MinIO/LocalStack usage.
Options
| Option | Default | Description |
|---|---|---|
fanout | 4 | Same-tier segment count that triggers a merge (size-tiered LSM) |
flushThreshold | 1000 | Docs in write buffer before auto-flush |
k1 | 1.2 | BM25 term-saturation parameter |
b | 0.75 | BM25 length-normalization parameter |
Errors
| Class | When thrown |
|---|---|
ManifestCorruptionError | manifest.json contains invalid JSON |
ManifestVersionError | manifest version is outside the supported range |
SegmentCorruptionError | CRC32 mismatch or missing segment file (.region tells you which) |
MappingCorruptionError | docids.snap or docids.log is corrupt |
TokenizerMismatchError | reopening an index with a different tokenizer config |
IndexLockedError | another process holds the advisory .lock file |
WriteStreamError | base class for streaming write failures (S3 multipart, etc.) |
Stats
| Method | Returns | Description |
|---|---|---|
docCount() | number | Documents indexed across all flushed segments |
segmentCount() | number | Number of active on-disk segments |
estimatedBytes() | number | Approximate in-memory footprint (postings buffers + sidecar arrays + Maps); lower-bound estimate for memory-budget callers |
Multi-writer / S3 safety
Termlog is single-writer per index directory. On local FS an advisory .lock file prevents concurrent opens in the same process group. On S3 (or any shared storage) there is no distributed lock — you must ensure at most one writer per index path.
License
MIT