@andwhy/e2e-pipeline

Scaffolds the full e2e testing pipeline into any project: agent-assisted Playwright tests (via Claude Code), CI runs against Vercel previews, and Checkly monitors that follow production deploys.

Quick start

pnpm dlx @andwhy/e2e-pipeline init

The CLI writes the config templates and workflows into your project and enables the e2e-pipeline Claude Code plugin for the repo — that plugin carries the agent definitions, the orchestration rules, and the guided skill, and Claude Code keeps it updated on its own. The install is purely additive — it never creates or overwrites CLAUDE.md.

The plugin lives in plugin/ in this repository, listed by the marketplace manifest at .claude-plugin/marketplace.json. It loads on developer machines (terminal, VS Code, desktop app); a claude.ai/code cloud session does not load repo-enabled plugins.

Setup and workflow docs: E2E Tests in Notion (why-agency internal). These are maintained in one place rather than scaffolded into each project, so a workflow change is published once.

Options

pnpm dlx @andwhy/e2e-pipeline init [--format ts|mjs] [--force] [--reset-config]

The agents, skill and rules update themselves through the plugin; the workflows through the reusable workflows they call. init --force is only needed for the package-owned files below, and init --force --update-workflows once to migrate a repo scaffolded before 0.14.0.

What gets created

From the plugin — nothing in the repo but two entries in .claude/settings.json (commit it):

• playwright-test-planner, playwright-test-generator, playwright-test-healer agents
• playwright-test-pipeline skill — guided walkthrough; triggers when someone asks to create or fix tests and escorts them through setup check, Plan → Generate → Heal, preview validation, and deployment. It also holds the project test rules and the agent prompt templates.
• A SessionStart hook with the always-on delegation rules (delegate to the agents, never hand-write spec bodies), active only in repos that carry the pipeline

Package-owned — regenerated by --force, so don't edit these by hand:

• .mcp.json — Playwright MCP server config (project root)
• playwright.mcp.config.{ts,mjs} — config the MCP server loads: extends playwright.config and clears testIgnore, so the planner/generator can execute seeds that normal runs ignore

Seeded once — starting points your project then owns. Written only when absent and never overwritten, --force included:

• playwright.config.{ts,mjs} — Playwright config template with BASE_URL support, testDir: "./tests/e2e", and testIgnore for seeds
• checkly.config.{ts,mjs} — Checkly config template with placeholders
• tests/e2e/seed.spec.{ts,js} — seed file for Playwright MCP setup_page
• tests/e2e/helpers/preview-consent-setup.{ts,js} — prepareVercelPreviewPage (Vercel deployment-protection bypass); projects extend it with consent logic
• .github/workflows/pr-playwright-tests.yml — runs the suite against the PR's Vercel preview and comments the result
• .github/workflows/checkly-deploy.yml — deploys monitors when a production deployment succeeds
• .github/workflows/checkly-heal.yml — a failing deployed monitor triggers a triage-and-heal agent: production regressions escalate to Slack immediately, with a minimal proposed fix as a draft PR when the cause is in the repo's own code; stale tests get fixed and flaky tests hardened — including behavior-neutral test hooks (data-testid) in app source when needed — and opened as a human-review PR, whose own preview run validates the app-side hooks. Inert until a Checkly webhook alert channel dispatches to it — the skill walks through that setup.

The three workflow files are callers of a few lines each. The work runs in this repository's .github/workflows/reusable-*.yml, referenced at @main, so a merge here updates every consumer's CI without a PR there. The heal prompt (templates/github/prompts/checkly-heal.md) and the PR comment script (templates/github/scripts/playwright-pr-comment.mjs) are read from the published npm package inside those workflows. One-time admin setting on this repository so other repos may call its workflows: Settings → Actions → General → Access → "Accessible from repositories in the organization".

The workflows need repository secrets: CHECKLY_API_KEY, CHECKLY_ACCOUNT_ID, VERCEL_AUTOMATION_BYPASS_SECRET, and for auto-heal SLACK_WEBHOOK_URL (values in 1Password). ANTHROPIC_API_KEY is an organization secret inherited by all repos — nothing to add.

E2E specs live under tests/e2e/ so a repo can keep unit tests elsewhere under tests/ without Playwright trying to run them. Seeds sit in the same tree and are kept out of test discovery by testIgnore rather than by living outside it.

The agents treat this as a default, not a rule: if a repo already has specs somewhere else, they write there instead. See rule 6 of the project test rules in the playwright-test-pipeline skill.

.gitignore gains Playwright/test output and seed entries; existing lines are left in place.

Scaffolded files are formatted with the project's own prettier when one is installed, so they match house style instead of this package's.

Agent customizations

These agents extend the vanilla Playwright test agents with:

• Extra tools: Generator gets terminal, edit, read_file; Healer gets terminal
• data-testid workflow: Generator reads recommended attributes from the plan and adds them to source components before recording
• Vercel share auth: every test.describe includes a beforeEach that authenticates with Vercel preview deployments
• Checkly verification: both Generator and Healer run pnpm exec checkly pw-test after local tests pass
• Numbered step comments: every logical step gets a comment (// 1. Navigate to…)
• Locale-awareness: no hardcoded domains or locale-specific URL patterns
• Optional draft input: Planner can validate a user-provided journey draft against the live page

The plugin

plugin/ is a Claude Code plugin in the standard layout: agents/, skills/playwright-test-pipeline/, hooks/, and .claude-plugin/plugin.json. agents/ is generated — npm run build:plugin inlines templates/claude/partials/_robustness-core.md into the agent sources under templates/claude/agents/ and syncs the manifest version from package.json. The output is committed because a marketplace is a git clone; CI fails a pull request whose output is stale (npm run check:plugin).

To try a branch before it is released, in any consumer repo:

/plugin marketplace add why-agency/why-e2e-pipeline#<branch>

or start Claude Code with --plugin-dir ../why-e2e-pipeline/plugin.

Auto-heal in CI does not load the plugin (a headless run with an API key has no plugin sources): the workflow copies plugin/agents/playwright-test-healer.md out of the published package instead, which is why plugin/ ships in the npm package too.

Documentation

The Installation and Usage guides live in docs/notion/ and are the source of truth. Merging a change to main publishes them to their Notion pages, stamped with the package version they document. Edits made directly in Notion are overwritten on the next publish.

Publishing needs a NOTION_TOKEN repository secret — a Notion PAT, or an internal integration token whose integration has been shared with both pages. Pull requests run node scripts/publish-notion.mjs --dry-run, which converts without writing, so malformed markdown fails before merge.

The stamp shows the latest version published to npm, not the version in package.json. Docs describing an unreleased version send people looking for features they cannot install yet.

Releasing

Bump version in package.json, add a CHANGELOG.md entry, merge to main. That's the whole release.

.github/workflows/release.yml compares package.json against the version on npm. If they differ it publishes and tags the commit; if they match it exits quietly, so ordinary merges publish nothing. It needs an NPM_TOKEN repository secret with publish rights on the @andwhy scope.

Published without npm provenance, which requires a public source repository.

License

MIT