testforge

AI-powered QA automation. Point it at a project and it scans the code, drafts a journey doc for each feature, and generates seeded Jest and Playwright tests that match the project's own structure.

Works in any project, with any AI CLI (defaults to Claude).

Install

npm install -D @activate-core/test-forge
```

testforge calls a local AI CLI to do its work. By default it uses
`claude -p`, so make sure the Claude CLI is installed and logged in.
To use a different CLI, see [Configuration](#).

## Quick start

From your project root:

```bash
testforge setup      # one-time: inspect project, map it, draft journey docs
testforge generate   # generate tests + seed files from the docs

After setup, review the drafted docs in docs/ and fix anything the AI got wrong — they are the source of truth for the tests. Then run generate.

When your code changes later:

testforge update     # re-check and refresh only what changed

Working on one journey

update is the whole-project command: if the code has moved it re-maps, re-drafts every doc and regenerates. Most of the time you want one journey, and both steps take --journey (or --area):

# the spec for one journey changed — re-draft just that doc, then its tests
testforge discover --journey payment
testforge generate --journey payment

# adding a journey that the reference now describes
testforge discover --journey refund-flow
testforge generate --journey refund-flow

# everything in one area
testforge generate --area ops

A feature that arrives on its own document

A ticket turns up with a PDF, or a spec for one feature. Point discover at it directly — you do not name the journey, the AI reads the document and works out which journey it describes:

testforge discover --reference ./docs/tickets/zh-4521.pdf

A reference only writes the journeys it actually covers. The ticket's PDF has never heard of your other journeys, so their docs are left exactly as they are, and the project's own source of truth stays where it was. TestForge then remembers that this one journey came from that document, and refreshes it from there on every later update — not from the project spec, which still does not describe it.

When a reference really is the whole product's new specification, say so:

testforge discover --reference ./docs/prd-v2.md --replace

A journey is its doc: generate reads docs/ and treats every markdown file there as one journey. So a doc you write by hand — for a feature the reference does not cover yet, or one taken from a ticket — works exactly like a drafted one. Write the doc, then generate --journey <name>.

If you edited a doc yourself, skip discover entirely and go straight to generate --journey <name>.

Regeneration replaces: when a journey is regenerated, test files for checks that no longer exist are deleted — but only files TestForge wrote. Your own tests are never touched.

Commands

Advanced (run a single step)

Flags worth knowing

When the reference cannot be checked

update compares two things: a hash of your source code, and a hash of the reference the docs came from. A reference it can read (a file, or a plain URL) is compared directly, so a spec that moved while the code stood still is detected — that case used to pass silently as "everything is up to date".

Some references cannot be read at all. A Notion or Confluence link answers a plain fetch with the application, not the document: no page content, and a build stamp that changes on every vendor deploy. Hashing that would be worse than not checking — it would report "unchanged" while the spec moved, and "changed" every time the vendor shipped. TestForge refuses it and tells you what it could not check:

No source changes detected.

  Note: the reference could not be checked — app.notion.com serves an app
  shell, not the document — its content needs the API.
  If the specification has changed, run `testforge update --refresh-docs`.

How it works

setup prepares the project in three stages:

1. init — detects the layout (monorepo or single app, where tests live, JS or TS) and writes testforge.config.json.
2. map — writes testforge.md: a short list of journeys and a summary of the data layer (database, models, key fields). The AI reads this instead of re-scanning all the code each time.
3. discover — writes one rich doc per journey into docs/ — fields, actions, expected results, edge cases, security notes, and the data a test needs.

generate turns each doc into a test. A journey with Type: api becomes a Jest test (HTTP + database checks); Type: web becomes a Playwright test. Other areas use the framework set in config. It also generates a seed file that provisions the test data each test needs.

generate hands each file it writes to the project's own ESLint with --fix before moving on. Formatting differences are corrected in place — CI runs lint before it runs a test, and a file that is correct but indented differently still fails the build. Anything the linter cannot fix (an unused declaration, say) is reported as a warning against that journey, because a declaration nothing checks is worth a person's attention. If the project has no ESLint, the step is skipped.

update hashes your source code AND the reference the docs came from, and compares both to the last run. Neither changed means no work and no AI calls; either one changed triggers a refresh, and it says which. A reference it cannot read is reported as exactly that, never as "up to date".

Source of truth

The journey docs in docs/ are the source of truth for the tests — generate reads the docs and nothing else. So where the docs come from decides everything.

By default the docs are derived from your code (the AI reads the project, writes testforge.md, then drafts the docs from it). That is fast, but it inherits whatever the code does: if a flow is buggy or a scenario is mishandled, the AI faithfully documents the wrong behavior, and the tests then lock that wrong behavior in.

So when the docs are drafted — during testforge setup (or testforge discover) — testforge asks what they should be based on:

What should the journey docs be based on?
  [1] testforge.md (what the AI derived from your code) (default)
  [2] An external reference (a file/PDF in scope, a website, or a Notion link)

Choose [2] to give testforge an authoritative reference of the expected behavior — a spec or requirements doc in the repo, a PDF, a website, or a Notion page. testforge then:

1. reads the reference and compares it against testforge.md,
2. writes a full comparison to testforge/reference/diff.md and prints a short summary of the differences,
3. asks you to confirm, and on confirmation drafts the docs from the reference — expected behavior comes from the reference, while routes, fields, models, and env var names come from the code/map,
4. generate then turns those docs into tests as usual.

The reference becomes the source of truth — expected behavior from a human or client, not just what the code happens to do. The choice is remembered in testforge/source.json, so a later testforge update refreshes the docs from the same source without asking again. Your review of testforge/reference/diff.md stays as an audit of what differed and why.

Non-interactive use (CI, scripts):

testforge setup --source map              # base the docs on testforge.md, no prompt
testforge setup --reference ./spec.md     # base the docs on a reference, no prompts
testforge setup --reference https://…     # a URL or Notion link works too

The same flags work on testforge discover. With no flags and no terminal attached, testforge does not block — it falls back to testforge.md and prints a note.

Running the generated tests

testforge generates the tests and seed files — you run them with your own test runner, the way your project already does. testforge does not run tests for you; that keeps it out of your test setup and CI.

A typical run:

1. Provide test credentials. The tests and seeds read the database URL and any secrets from environment variables — never hardcoded.

   testforge setup writes a .env.testing.example for you. It lists every env var the journeys need, grouped by area (api / web / ops), and annotates each one with the journeys that use it — so you know exactly which credentials a feature needs, and can skip the ones for features you don't run. For example:

   # ========================================
   # API
   # ========================================
   DB_URL=              # used by: (all)
   TOKEN_SECRET=        # used by: auth, user
   STRIPE_SECRET_KEY=   # used by: payments, subscription

   # ========================================
   # WEB
   # ========================================
   StripePublicKey=     # used by: payments

Copy it to .env.testing, fill in real values, and keep that file out of version control (add it to .gitignore). Re-running setup or update refreshes the example as your code changes.

Note: some credentials aren't static — e.g. a user access/refresh token from a login flow. Those belong in your own test setup (a global setup or seed step), not in the env file.

Keep this file out of version control (add it to .gitignore). The exact variable names come from your own code — the journey docs and the project map list the env var names each journey needs.

2. Run the seeds to put known test data in the database. Run the seed files in seedOutput (see config) with your runtime, e.g.:

   npx tsx path/to/seed/<journey>.seed.ts   # or: node <journey>.seed.js

3. Run the tests with the project's own command, e.g.:

   # api (Jest)
   NODE_ENV=testing npx jest

   # web (Playwright)
   npx playwright test

The test runner produces its own pass/fail report.

In CI (GitHub Actions)

npx testforge ci --github                       # run only when started by hand
npx testforge ci --github --schedule weekdays   # also Mon–Fri 06:00 UTC (daily, weekly or a cron line too)
npx testforge ci --github --env-name staging --force

The workflow runs testforge run when started by hand (and on the schedule, if given), against a deployed environment, and publishes each run to the dashboard. It reads its values from repository secrets with the same names as your env file (TestForge reads only the names, never the values), plus TESTFORGE_KEY, and the dashboard address from the repository variable TESTFORGE_DASHBOARD_URL. Point the base URL secrets at the deployed app. Test results are kept as a workflow artifact for 14 days.

Configuration

testforge setup writes testforge.config.json for you. You can edit it:

{
  "aiCommand": "claude -p",
  "testOutput": {
    "api": "packages/api/test",
    "web": "packages/webapp/tests",
    "ops": "packages/ops/tests"
  },
  "framework": {
    "api": "jest",
    "web": "playwright",
    "ops": "playwright"
  },
  "seedOutput": "packages/api/test/helpers/seed",
  "testExtension": ".test.ts",
  "externalModules": ["shipengine", "@customerio/cdp-analytics"]
}

• aiCommand — the CLI used to call the AI. Any tool that reads a prompt from stdin and prints the reply works, so testforge is not tied to one provider. testforge appends the right flags per phase: the setup phases (init/map/discover) run with edit permission because the model writes files, while the generation phases (tests/seeds) run with no tools (--allowedTools "" for Claude) so the model just prints code instead of trying to create files — which would otherwise leak "the write wasn't approved…" prose into the output.
• testOutput — where tests go, per area. Areas are not fixed — use whatever your project has (api, web, ops, lambda, ...).
• framework — which test framework each area uses.
• seedOutput — where seed files go.
• testExtension — .test.ts or .test.js.
• externalModules — the third-party network services this project talks to, named as a test would import them. A generated backend test that imports one without mocking it is rejected before it is written: left alone it would reach the live service, so it breaks offline and can fire real charges, messages or shipments. testforge already knows the common SDKs (payment, SMS, email, cloud, identity) and this list is added to them, never instead of them — so name only what is particular to your product. Every domain has something: a shipping carrier, a CMS, a ticketing or streaming backend, an ad network.
• relatedDirs — other repos this project depends on, read-only, e.g. ["../backend"] for a frontend whose API and database live next door. Every AI phase may read them, so the map captures the real data layer and tests can check what the API stores. A headless run cannot read outside the project otherwise.
• decisions — a file or link listing what is NOT a requirement yet: open product decisions and known issues left out on purpose. Docs do not turn them into criteria, and a failing test that falls under one is reported as a pending decision, never as a defect. Keeps the reference you share with a client clean while the tests still know what is undecided.

How generate keeps the AI cost down

The journey doc is read once and kept as the journey's memory (testforge/memory/<area>/<id>/): its rules with their source, level, tests and history, the plan, and the code facts. No later step sends the whole doc again.

1. Rules. The doc is split into rules (R1, R2 …) without AI. Rule ids stay stable when the doc changes. Every rule records its source, the files that test it, why it was skipped, and its earlier wordings.
2. Plan, without AI. Each numbered section of the doc becomes one test file ("planner": "ai" asks the AI instead). Rules another journey already tests are not planned again. The plan check (plan-check.md) lists every rule that gets tests, every skip with its reason, and any must rule left out.
3. Code facts, once. The source the doc points at (the files it cites, the components its controls are named after, the pages and hooks that reach them) is collected without AI and summarised in one call.
4. Writing. Each call writes a few files (filesPerCall, default 3) and gets only their rules, their controls' facts and the testkit's exports; the shared rules for writing a test are a cached system prompt. Tests find every element with kit.find(page, kind, name), check outcomes rather than markup, and stub a state they cannot create.
5. Checks before anything runs. A file is refused, and written once more with the reason, if it sleeps for a fixed time, skips because an element is missing, has no test or no assertion, spends more than maxTestsPerRule tests on one rule, or names a label found neither in the code nor the doc.
6. Every rule, a test that can fail. Every scenario comes from the doc (plus the edge, security and user cases TestForge adds); what the code has or lacks never removes one. Each test is checked on its own: every planned rule has a test, or — only when it needs an outside service the test cannot drive (a payment provider's inside, an email inbox, another app) — a // cannot-test: R7 — <the service> line; no assertion accepts the good and the bad outcome alike, compares the page's whole text, sits only inside an if, or repeats what its own stub returned; kit.find is awaited and finds elements by the names the screen shows; kit.openStep names a step the testkit has. The tests that fail a check are taken out and their rules written again in ONE small call; what is still wrong is removed and reported. The summary line says how many planned rules have a test, which cannot be tested here and why, and which have none, and rules.json / plan-check.md record the same.
7. A changed doc. Only what changed is touched: a renamed label is replaced in its tests, a removed rule's tests are deleted (both without AI); a reworded or new rule gets its tests written by one small call.

Names the product gives. A test reaches the product through names: a route, a button's label, a command, a table or field. When the code already has one, the test uses it. When it does not (yet), the test asks for it by a plain name — need('order summary URL') from @activate-core/test-forge/reporters/interface.cjs — and the name is listed in testforge/interface.json with an empty value. The developer fills it once it exists; testforge run and bind hold a file back, and list what it waits for, while one of its names is empty. TestForge never overwrites a value.

Every run ends with its AI cost per step (write, grounding, seed, harness …).

A feature that is not live yet. Put Status: not built or Status: behind flag <name> in the journey doc's header. Every file of the journey is stamped // testforge-pending: …, and testforge run (and bind) hold it back until the line is removed; --all runs it anyway. No seed is written for it: its tests make their own data.

generate runs nothing. testforge run runs the suite; testforge review judges failures when you ask for it.

Every Claude call reports its real token use and cost; TESTFORGE_TRACE=1 also records each call's prompt and events under testforge/runs/trace/.

Project state

Everything TestForge keeps lives in testforge/. Commit it, except runs/ (run output and AI logs): a teammate who pulls it starts from your memory, plans and code facts instead of paying for them again. A project set up by an older version is moved from test-artifacts/ on the first command.

Notes

• Generated tests and docs are a strong draft, not final. Review them. Where something could not be established, a test is pending, with the reason in its annotation (and in the dashboard).
• Secrets are never written to docs, configs, or tests. Connections use environment variable names only; real values come from your environment at run time.

License

MIT