行动日志-s3
Amazon S3 存储后端 @ backloghq/ opslog 页面存档备份,存于互联网档案馆。将您的事件源数据存储在 S3 中,并允许多写机的货币.
安装
npm install @backloghq/opslog @backloghq/opslog-s3
使用量
import { Store } from "@backloghq/opslog";
import { S3Backend } from "@backloghq/opslog-s3";
const store = new Store<{ title: string; status: string }>();
await store.open("my-store", {
backend: new S3Backend({
bucket: "my-bucket",
prefix: "opslog/my-store",
region: "us-east-1",
}),
agentId: "agent-A",
});
await store.set("task-1", { title: "Build API", status: "active" });
const task = store.get("task-1");
await store.close();
另一台不同机器上的代理可以写给同一商店:
const store = new Store<{ title: string; status: string }>();
await store.open("my-store", {
backend: new S3Backend({
bucket: "my-bucket",
prefix: "opslog/my-store",
region: "us-east-1",
}),
agentId: "agent-B",
});
// Sees agent-A's writes
store.get("task-1"); // { title: "Build API", status: "active" }
// Writes to its own WAL — no contention with agent-A
await store.set("task-2", { title: "Write tests", status: "active" });
await store.close();
选项
new S3Backend({
bucket: "my-bucket", // S3 bucket name (required)
prefix: "path/to/store", // Key prefix (optional, no trailing slash)
region: "us-east-1", // AWS region (required if no client provided)
client: myS3Client, // Pre-configured S3Client (optional)
lockTtlMs: 60000, // Lock TTL in ms (default: 60000)
});
预配置客户端
自己传 S3Client 如果您需要自定义配置( 证书、 终点、 中间软件) :
import { S3Client } from "@aws-sdk/client-s3";
const client = new S3Client({
region: "us-east-1",
endpoint: "http://localhost:9000", // MinIO, LocalStack, etc.
forcePathStyle: true,
});
const backend = new S3Backend({ bucket: "my-bucket", client });
如何运作
存储布局
s3://bucket/prefix/
manifest.json # Store metadata
snapshots/snap-<ts>.jsonl # Immutable state captures (JSONL)
ops/agent-<id>-<ts>.jsonl # Per-agent WAL files
archive/archive-<period>.json # Archived records
WAL 操作( 每批对象)
S3没有本地附件。 S3Backend 不再下载,并重新加载一个正在增长的单一文件,而是将每个文件作为单独的S3对象存储:
ops/agent-A-1744200000/
batch-1744200001-0000.jsonl (ops from first write)
batch-1744200002-0000.jsonl (ops from second write)
batch-1744200003-0000.jsonl (ops from third write)
- 附录: 单发对象( 不需要下载)
- 读取: List ObjectsV2 + 平行 Get Object 每批, 有增量缓存( 在后续读取时只下载了新批)
- 截断: 删除上一个批次对象(或如果包含多个操作则修改)
用 v. 1. 0 (单) 创建的存储 .jsonl 文件)是透明支持的——后端自动检测出格式.
锁定
使用 S3 有条件的写法( U)IfNoneMatch: *)用于锁定获取——如果锁定已经存在,则在解剖上失败. 被锁的物体携带TTL;从坠毁的特工中自动取回被困的锁.
Blob 存储 + 字节范围读取
await backend.writeBlob("data/records.jsonl", buffer);
const full = await backend.readBlob("data/records.jsonl");
const range = await backend.readBlobRange("data/records.jsonl", 1024, 256); // HTTP Range header
readBlobRange 使用 S3 GetObject 与 Range: bytes=N-M 页眉 — 只读取请求的字节 。 在JSONL唱片商店中启用O(1)点取景,而无需下载完整对象.
解决冲突
当多个代理给同一密钥写信时,OPslog的Lamport时钟命令通过最后的写作胜出解决冲突. 见 操作日志多写文件 详细情况.
IAM 权限
S3Backend 在桶/前缀上需要这些 S3 权限 :
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::my-bucket",
"arn:aws:s3:::my-bucket/opslog/*"
]
}
发展
npm run build # Compile TypeScript
npm run lint # ESLint
npm test # Run tests (uses in-memory mock S3, no AWS needed)
npm run test:coverage # Tests with coverage
许可证
麻省理工学院