hollow-test

ADLC phase: P3 gate (C4 — diff-scoped mutation)

Diff-scoped mutation gate — the honest coverage check. Mutates only the lines changed in your diff, runs your test suite against each mutant, and fails if any mutation survives. A surviving mutant proves hollow coverage: lines are executed but their behavior is unconstrained by any assertion.

Diff-scoping keeps the run at seconds-to-minutes rather than the hours that kill whole-codebase mutation testing.

Usage

hollow-test --test-cmd "node --test test/" [options]

Flags

Exit codes

--max too small for the diff: a starved file fails closed

--max spreads the mutant budget round-robin across every selected file, and a small --max against a wide diff can leave one or more diff-derived files with zero quota — never mutated at all. That file's changed code is then unverified, so hollow-test refuses to report a pass: it exits 1 and names every starved file on stderr rather than silently passing on whatever other files did get budget. Raise --max to cover every selected file (one per file changed, at minimum), or narrow the diff:

$ hollow-test --test-cmd "node --test test/*.test.mjs" --base HEAD~1 --max 1
error: 2 selected file(s) received no mutation budget and were NOT prosecuted: src/b.mjs, src/c.mjs — raise --max to cover them.

This does not apply to a file that received budget but has no mutable line at all (a comment-only or import-only change) — that case still only warns, since the file genuinely has nothing to mutate rather than having been skipped for lack of budget.

--target / --rails: the P3 rails-authoring / characterization-test case

A diff that adds only test files — exactly the shape of a P3 rails-authoring ticket, or a characterization-test ticket that pins existing, unchanged (frozen-rail) behavior — has nothing in filterTargetFiles()'s diff scope to mutate. Rather than silently reporting a vacuous 0/0/0 pass (indistinguishable from a genuinely strong suite), hollow-test exits 1 in that case unless an explicit target is given:

# Diff is test-only (new rails for src/foo.mjs, which itself didn't change) — mutate
# src/foo.mjs directly so the new rails are actually prosecuted:
hollow-test --test-cmd "node --test test/foo-rails.test.mjs" --base main \
  --target src/foo.mjs

# Same, but read the target from a ticket's declared "rails" (single-ticket object or a
# full tickets.json — rails merged across all tickets in the file):
hollow-test --test-cmd "node --test test/foo-rails.test.mjs" --base main \
  --rails .adlc/tickets.json

Examples

# Check the last commit
hollow-test --test-cmd "node --test test/" --base HEAD~1

# Check staged changes vs main
hollow-test --test-cmd "npm test" --base main --max 30

# Machine-readable output for CI
hollow-test --test-cmd "node --test test/*.test.mjs" --json

Safety guarantees

1. Dirty-tree check: refuses to run if git status --porcelain is non-empty. This prevents accidentally leaving a corrupted file if the process is interrupted. Commit or stash your changes first.

2. File restoration: every mutated file is restored via a try/finally block — even if the test command crashes or the process is interrupted via SIGINT. The SIGINT handler performs an emergency restore before exiting.

3. Sequential execution: mutants are applied and tested one at a time (never in parallel) to avoid concurrent writes to the same file.

4. Green baseline: requires a green baseline before mutating so a failing suite cannot masquerade as killing every mutant. If the baseline suite fails, hollow-test emits the test command's captured stdout and stderr to stderr before exiting, ensuring failure reasons and failing tests are diagnosable.

5. A timeout ends everything the suite started: each trial runs through a small watchdog child (lib/watchdog.mjs) that keeps the suite in your own process group and session and, when the timeout fires, freezes the victims (SIGSTOP, root outward, re-collected until stable, so a fork during the kill cannot slip out by being reparented) and then SIGKILLs them. Victims are the shell's live descendants plus, on Linux, every process whose environment still carries the marker the shell was started with (a chain, so a nested hollow-test extends it rather than hiding its subtree from the outer sweep), which catches a helper the suite double-forked or detached earlier in the trial. Anything ended beyond the shell itself is named on hollow-test's stderr as it happens (and folded into that trial's captured output), so a daemon the suite detached does not vanish silently. Parent links come from /proc on Linux with no fork, so this works on the memory-starved host it exists for; other platforms use a time-boxed ps and the live tree only. The watchdog also polls its parent: if hollow-test itself is killed outright (kill -9, a caller's timeout), it ends the suite and exits rather than being stranded with it. The same sweep runs on Ctrl-C (a shell starts & jobs with SIGINT ignored, so a helper started as server & npm test would otherwise survive it) and whenever the shell exits on its own, so a crashed node --test cannot strand a worker any more than a timeout can (on that path the shell is already reaped, so only the marker sweep can still name what it left, on Linux). What still escapes: a process that scrubs its own environment after detaching, and a detached helper on a platform without readable /proc. Before all this, node --test, its file workers and anything the suite spawned outlived every timed-out mutant as orphans, still running the mutant code; a pile of them took a host down (2026-10-08: RAM and swap full, nothing killed, SSH logins that never got a shell). Windows runs the command unwrapped.

6. Every node gets a V8 heap cap: each node the test command starts gets --max-old-space-size via NODE_OPTIONS (default 2048 MiB, inherited by the suite's own child processes), so a mutant that makes JS-heap growth unbounded dies with "heap out of memory" instead of running until the timeout. This is a per-process bound on one isolate's old space, not a host bound: node --test runs files in parallel, and Buffer, ArrayBuffer and native memory are outside it (the 2026-10-08 runaway was an endless stream of Buffers). Bound the host with a cgroup (systemd-run --scope -p MemoryMax=…); rely on guarantee 5 to end a runaway. HOLLOW_TEST_MAX_OLD_SPACE_MB=<n> changes the cap; 0 disables it; when the variable is unset, a --max-old-space-size (or --max_old_space_size) you already put in NODE_OPTIONS is kept; when it is set, it replaces any inherited flag. The baseline run uses the same settings, so a suite that genuinely needs more fails loudly there, not silently mid-gate, and the stderr it prints names the cap and the variable. The default is below Node's own default on large hosts; a suite that needs more sets the variable once.

What is mutated (and what is skipped)

Mutation applies to plain JavaScript only: .mjs, .cjs, .js. This is an allow-list. TypeScript and JSX (.ts, .mts, .cts, .tsx, .jsx), Python, CSS and everything else are excluded, because the operators are text-based and cannot tell a comparison from a type argument or a JSX delimiter — Promise<unknown> becomes the invalid Promise>=unknown>, and a parse failure is currently scored as a killed mutant. See issue #293.

A file is also skipped when a path segment is test, tests, spec, specs, or __tests__, or when its basename matches a node --test discovery convention: test.js, test-*, test_*, *-test.*, *_test.*, *.test.* (and the spec equivalents).

Matching is segment- and basename-anchored on purpose: a substring test would exclude production paths such as packages/hollow-test/lib/targets.mjs or lib/attest.mjs.

Two escape hatches, because no convention resolves every case:

• --test-glob <glob> — treat additional paths as tests.
• --source-glob <glob> — treat paths as production source even when their name matches a test convention. Needed for product names like hollow-test.mjs and spec-lint.mjs, which are indistinguishable from tests by naming alone. For the same reason, hyphenated forms (foo-test.js, spec-foo.js) are not treated as tests — a hyphen is ambiguous between a test convention and a product name, and hollow-test.mjs and spec-lint.mjs are production files. If you use that convention, keep tests in a test/ directory or name them *.test.*. --target/--rails files bypass the test-path exclusion — the caller is deliberately naming a mutation target, and rails are usually test files.

They do not bypass the language allow-list. Operators are JS/TS-shaped, and a mutant that renders another language syntactically invalid makes the test command exit non-zero, which is scored as killed — a false pass. So:

• --target <path> in an unsupported language is refused (exit 1). The caller named one file; dropping it silently would be its own silent pass.
• --rails <ticket> expansions legitimately match non-source (schemas/**, JSON, fixtures). Those are filtered out and reported, and the run fails only if nothing mutable remains.

Within diff-derived eligible files, only lines changed in the diff are targeted; --target/--rails files are mutated in their entirety. Lines that are blank, comments, imports, export {, or console.* calls are skipped.

Comment-only changes are reported as not covered

A changed file whose change leaves its program untouched — only comments or blank lines added, reworded or removed — has no behaviour to mutate. It is not failed; it is listed on stdout as not covered, and when it is the only change the run exits 0 with comment-only diff — no changed behaviour to mutate. With --json the same files appear under "skipped": { "commentOnly": [...] }, on that early exit and in the normal report alike.

Both sides of the diff are judged, so these are not comment-only and stay mutation targets (failing closed when nothing mutable remains):

• a code line deleted and replaced by a comment or a blank line;
• live code wrapped in an added /* … */;
• a deleted */ that turns the code after it into comment text;
• a // ... line inside a multi-line template literal, which is string data.

Any doubt — an unreadable file, an old side that cannot be read from --base, a scan that loses its place — keeps the file a target.

Invalid mutants

A mutation that produces code Node cannot parse is discarded, not scored. Line-based operators produce these routinely — null-return rewrites a multiline return { to return null; and strands the object literal's remaining lines.

This matters because a kill is inferred from a non-zero exit, and a file that does not parse also exits non-zero. Counting such a mutant as killed fakes coverage; counting it as survived blames the tests for code that was never valid. It is reported in its own invalid bucket in both the table and JSON.

If every mutant in a run is invalid, hollow-test exits 1 (operational failure) rather than passing: no assertion was exercised, so the run proves nothing.

Validation uses node --check, so no parser dependency is added and the real file extension and package type are honoured. Both the syntax check and the test run are tri-state — valid, invalid, or unknown. "Could not determine" never collapses into "valid": if the checker is killed, times out, or cannot spawn, the run fails operationally rather than guessing. The same applies to the test command itself: a spawn failure (EAGAIN, ENOMEM) is not a timeout, and a timeout is the only non-completion that counts as a kill. A kill must mean the tests ran and failed. Assuming validity would run the test command against unparseable source, whose non-zero exit is then scored as a kill — the very path this closes.

The all-invalid guard is applied per file, not just globally. A global check passes the moment any other file yields a kill, which would let an explicitly named --target go entirely untested while the run reports success.

Mutation operators (from @adlc/core)

off-by-one does not mutate a tuning constant. +1 on a duration in milliseconds or a size in bytes is an equivalent mutant — no test can observe 60000 -> 60001 ms on a subprocess timeout — so generating it would demand a test that cannot exist (#359). The rule reads the constant's name, split into segments, and snake_case/camelCase spell the same thing:

A trailing count word always wins over a tuning phrase earlier in the name, so widening the phrase list can never silently delete prosecution of a count. Naming the unit is what earns the mask — MAX_BUFFER_SIZE_BYTES is masked where MAX_BUFFER_SIZE is not (#372). | negate-guard-subclause | Array.isArray(x) ↔ !Array.isArray(x); if (value) → if (!value); loose v == null ↔ v != null | | array-literal-shrink | ['id', 'title', 'scope'] → ['id', 'title'] | | ternary-swap | cond ? a : b → cond ? b : a |

JSON output schema

{
  "tool": "hollow-test",
  "summary": {
    "total": 5,
    "killed": 4,
    "survived": 1
  },
  "mutants": [
    {
      "file": "src/calc.mjs",
      "line": 7,
      "operator": "null-return",
      "status": "survived",
      "timedOut": false,
      "original": "  return a + b;",
      "mutated": "  return null;"
    }
  ],
  "skipped": { "commentOnly": ["src/notes.mjs"] }
}

skipped is present only when a changed file was left out as comment-only.

Relationship to sibling tools

• rails-guard (C5): enforces that test files are not modified during build (they are the measuring instrument). hollow-test verifies that those tests actually constrain behavior.
• review-calibration (C8): uses the same mutate operators to plant bugs and measure reviewer recall. hollow-test and review-calibration share core mutation machinery.
• flail-detector (C6): hollow-test is a P3 gate; flail-detector watches the P4 build session. They serve complementary phases.

Core gaps

None. All required functionality (gitDiff, isDirty, isGitRepo, git, globMatch, mutate.generateMutants, mutate.applyMutant, mutate.changedLinesFromDiff, parseArgs, pass, gateFail, opError, printJson) is available in @adlc/core.

Implementation notes

NODE_TEST_CONTEXT stripping

Node.js v22 sets NODE_TEST_CONTEXT in child process environments when running under node --test. If a child process inherits this variable and itself calls node --test, it silently skips all test files (exits 0). hollow-test strips NODE_TEST_CONTEXT from the child environment before running each mutant's test command. This ensures mutation trials work correctly even when hollow-test is itself running inside a test harness.