@draht/rlm

Recursive Language Models (RLM) — the inference-time scaffold from Zhang, Kraska, Khattab (2026), Recursive Language Models (arXiv:2512.24601): root-LLM-produces-code → REPL-executes → truncated-stdout → history-append → FINAL-check. Lets a root model answer questions about inputs far larger than its own context window by writing Python that inspects, chunks, and searches the content from inside a sandboxed REPL, optionally recursing into sub-llm_query calls, instead of the caller ever loading the raw content into a single prompt.

When to prefer RLM over a normal agent read

Reach for RLM when the input genuinely doesn't fit the model's context window, or fits only by truncating away content that might matter:

• A log file, dataset dump, or exported knowledge base that's tens or hundreds of megabytes — orders of magnitude past any model's window.
• A directory/glob of source files too large to read whole without evicting earlier conversation context.
• Any case where you'd otherwise have to guess which prefix/suffix/sample of the input to truncate to, and a wrong guess means missing the answer.

Don't reach for RLM when the input already fits comfortably in the model's context window — a normal read is faster, cheaper, and easier to debug. RLM trades one direct read for several extra model round trips (see below); that trade only pays off when the alternative is truncation-induced information loss, not just "this file is somewhat long."

Cost envelope

A normal read is one call: the whole input goes into the prompt, the model answers once. RLM is not universally cheaper than that — it's a different trade:

• The root model makes one call per REPL step (a handful to a few dozen steps for a typical run), and each step's Python may itself call llm_query(...), which is a second model call (typically to a smaller "rlm-sub" role — see router-session.ts's RlmRole) that also costs money.
• So RLM's total cost is sum(root-step costs) + sum(sub-call costs) versus a baseline's single call. RLM wins when the baseline's single call would otherwise require either failing outright (input doesn't fit at all) or paying for a much larger context window / re-reading the input multiple times across a conversation to compensate for truncation — not in the general case.
• Don't guess — measure. src/cost-comparison.ts's compareCost() computes both numbers for one trajectory: rlmCostUsd from the trajectory's own logged cost entries (readCostLog/getSessionCosts from @draht/router, filtered by trajectoryId), and baselineCostUsd as the estimated cost (via @draht/router's estimateCost) of a single call that truncates the same content down to the target model's context window (first-N-characters-that-fit — a simple, named strategy, not a claim that it's the smartest possible truncation). Run it against a real trajectory ID before assuming RLM is the cheaper (or more expensive) option for your specific input/model pair; .draht/rlm/cost-comparison-report.json is where a comparison run writes its report.

Observability: trajectory logs & replay

Every createRouterBackedSession run writes a newline-delimited JSON trajectory log to .draht/rlm/<trajectoryId>.jsonl (src/trajectory.ts) — one TrajectoryStepEntry per REPL step (its code, truncated stdout, error if any, the sub-calls it triggered, and that step's own cost) plus one terminal TrajectoryFinalEntry once the session resolves. trajectoryId is the same ID already used to tag the session's @draht/router cost-log entries, so a trajectory's logs and its cost accounting always correlate by the same key.

Because the whole run is on disk, it can be reconstructed with zero LLM calls:

draht rlm replay <trajectory-id>

reads the JSONL log via readTrajectory() and prints the final answer (add a verbose flag for the full step-by-step trace) — no router, no model, no network access on that code path at all. Useful for auditing what a session actually did, debugging a run that didn't reach FINAL, or reviewing cost without re-running anything.

The test/s-niah.test.ts regression suite exercises recall at 10x and 100x a mocked root model's context window (synthetic needle-in-haystack at ~4M and ~40M characters, against a scripted mock rootLlm — no real model calls) as a standing check that the scaffold still finds a planted needle at scale.

Worked example

Say deploy.log is a 40 MB deployment log and you need to know what caused an outage buried somewhere in it — too large to paste into a normal prompt. The same question, three ways:

Interactive /rlm command (inside a draht session):

/rlm deploy.log What caused the outage around the 03:00 UTC deploy?

draht rlm CLI (scriptable, one-shot):

draht rlm --input ./deploy.log --query "What caused the outage around the 03:00 UTC deploy?"

rlm_query tool (an agent defers the read itself, mid-session, instead of loading the log into its own context):

{
  "tool": "rlm_query",
  "input": {
    "input": "deploy.log",
    "query": "What caused the outage around the 03:00 UTC deploy?"
  }
}

All three share the same path under the hood: parseInputArg/loadInput resolve deploy.log into content, createRouterBackedSession wraps it as the REPL's context with the question prepended as a header, and the root model writes Python to grep/chunk/search context — recursing into llm_query sub-calls for individual chunks as needed — until it calls FINAL(...)/FINAL_VAR(...) with the answer. --input/input also accepts a glob ("src/**/*.ts"), an http(s):// URL, or knowledge:<client-slug> for a named client knowledge base.

To check whether that particular run was worth it cost-wise, take the trajectoryId of its .draht/rlm/<trajectoryId>.jsonl log (or the matching trajectoryId field in .draht/cost-log.jsonl) and feed it into compareCost() (see Cost envelope, above) rather than assuming either direction.

Sandbox requirements

Linux RLM sessions require unshare or bwrap, a 64-bit x86 or ARM Python interpreter, seccomp filtering, and an enabled Landlock ABI 3 or newer. The bootstrap grants read access to isolated Python module paths, system library directories (including the Nix store when present), the driver, and runtime support files. Regular-file writes are confined to the session workdir; /dev/null is also writable. Process execution, networking, and filesystem metadata changes are denied. File metadata such as names and sizes is not hidden.

If these kernel restrictions cannot be installed, the driver does not run. macOS continues to use sandbox-exec. Both platforms run a sandbox startup check before accepting model-generated Python.

The preferred unshare path keeps Python at the PID monitored for memory usage. With the bwrap fallback, RSS polling currently observes the wrapper process instead; the session's RSS ceiling is not reliable on that path.

Development

npm install          # from repo root, resolves this workspace package
cd packages/rlm
npx tsc --noEmit     # typecheck
npm test             # vitest

Package layout

• src/session.ts — RlmSession root loop (step/run, FINAL/FINAL_VAR sentinels, constant-size history).
• src/router-session.ts — createRouterBackedSession, the production @draht/router-backed factory (prompt-tier selection, cost logging, trajectory logging).
• src/trajectory.ts — trajectory JSONL append/read (this phase).
• src/cost-comparison.ts — RLM-vs-baseline cost harness (this phase).
• src/loaders.ts — --input//rlm <input> source loaders (file, glob, URL, knowledge:<slug>).
• src/prompts.ts — tiered system-prompt templates, selected by the root model's context window.
• python/repl_driver.py — the sandboxed Python REPL driver.
• packages/coding-agent/src/rlm-cli.ts — draht rlm CLI subcommand.
• packages/rlm-agent/ — /rlm command + rlm_query tool extension for normal agent sessions.

AGENTS.md

RLM is a cross-cutting coding-agent capability, not a property of any single generated project's stack — none of packages/templates/src/*.md's per-stack templates (Astro, Go/gRPC, SST TypeScript) is the right home for a "when to use RLM" note, and this repo's own root AGENTS.md is a contributor-workflow doc, not a feature reference. This README is the primary documentation surface for RLM; no template changes were made.