BenchRouter CLI
Use BenchRouter from a terminal. The CLI initializes a repository, checks its integration, upgrades the generated kit, and reads route evidence.
The package is @benchrouter/cli. It installs the benchrouter command.
npx @benchrouter/cli --help
Connect Databricks
Install the Databricks CLI and use an OAuth workspace-admin login:
benchrouter login
databricks auth login --host https://<workspace>.cloud.databricks.com
benchrouter databricks connect \
--host https://<workspace>.cloud.databricks.com \
--catalog workspace --schema <schema> --warehouse <warehouse-id>
Both a user OAuth login (U2M) and a service principal OAuth login (M2M) are supported. For M2M, configure the Databricks CLI with the admin service principal's client id and client secret. Connect reads the credentials into memory and obtains an access token from the workspace. It refreshes either login when fewer than five minutes remain. The client secret and login tokens never reach BenchRouter.
Connect needs an owner or admin role in the BenchRouter account. The catalog and
schema must exist. Omit --warehouse to use the only serverless warehouse.
With several warehouses, choose one on a terminal or pass its id. Add
--results-group <group> and --manage-group <group> to share results and management
with account groups. Use --json for a secret-free machine result. Existing
--api-url, --account-token, and saved login settings apply.
The Databricks login token stays in process memory on your machine. BenchRouter
receives only a fresh identity-only workspace proof. Connect writes the poll key
to Databricks before grants. A repeated Connect reuses the recorded SP and job ids.
It resets the job ACL. Under restrict_workspace_admins, the connecting admin
owns the job and the agent SP gets CAN_MANAGE. Removing that admin can stop the job.
Exit codes: 3 means a refused initial grant or heal and names the owners, 4 means
workspace_admin_required, and 5 means a PAT login. Use OAuth to replace a PAT.
Other errors exit with 1. Connect waits up to three minutes for a successful agent
run, binding verification, and five expected setup rows to exist and be done:
volumes <catalog>.<schema>.benchrouter_artifacts and
<catalog>.<schema>.benchrouter_results, results table
<catalog>.<schema>.benchrouter_eval_results, experiment
/Shared/benchrouter/<binding_id>, and eval job benchrouter-eval-<binding_id>.
The catalog and schema are this Connect's --catalog and --schema values.
Volume and results-table targets match without regard to case. Experiment and eval-job
targets must match exactly.
The binding read gives each object's current state. An older done row can satisfy
completion. A re-queued row reads pending and keeps Connect waiting.
Pending serving endpoints do not block completion. Any failed setup row stops the command
and reports its title, message, and owner. A customer owner requires a workspace
admin to fix it. A timeout names the expected rows that are missing or incomplete.
The JSON result includes setup_items, with kind, target, and state for each
item. Human output prints one line per item.
After grants and heal, Connect runs every pending SP-change grant, then every
pending revoke from the server's cleanup_pending. It acknowledges completed or
absent steps once. A refused SP-change step stays pending. Missing folders, manifests,
objects and tables count as done. An invalid manifest, reserved secret scope or
undeclared route keeps the grant or revoke pending with a warning. Its pending_steps
entry includes the reason. Fix the manifest and run Connect again.
Connect continues through job creation and verification and exits 0 when setup succeeds.
Permission warnings name the step and its owner. Re-run benchrouter databricks connect as that owner or a
metastore admin. JSON includes refused steps in pending_steps, with kind, id,
action, object, target, and owner. A retry resumes steps even when
sp_change is null.
Read workspace state, attach an eval, or disconnect:
benchrouter databricks status --host https://<workspace>.cloud.databricks.com --json
benchrouter databricks add-eval support/triage /Shared/evals/triage --host https://<workspace>.cloud.databricks.com
benchrouter databricks disconnect --host https://<workspace>.cloud.databricks.com
benchrouter databricks disconnect --host https://<workspace>.cloud.databricks.com --force --json
benchrouter databricks disconnect --host https://<workspace>.cloud.databricks.com --reclaim --json
All four commands support --json. Status is available to any account member.
By default, it lists open bindings and closed bindings with nonempty cleanup_pending.
Pass --all to show every matching binding, including archived and disconnected
bindings without leftovers. Text and JSON use the same selection. Each binding includes
state, check-in, cadence, agent SP, routes, exposures, eval folders, grants, setup items,
failures and the server's exact cleanup
instructions and commands. JSON always has the shape {bindings: [<server read>, ...]}.
Each entry is the full server read after any cleanup acknowledgement. With no
matching bindings, it returns {bindings: []}. Text shows agent and eval job
links to their Databricks job pages, where admins can inspect the last runs.
Status needs no Databricks login. When an OAuth login is available, status checks
pending workspace objects. It reports objects that are gone to the cleanup endpoint
and reads the binding again. An account member can still read status if the server
refuses that acknowledgement. Account-console SP steps stay pending; a workspace
SCIM read cannot prove that an account SP was deleted. Status shows pending
grant/revoke instructions, actions and targets in Leftovers. It never runs or acknowledges those steps. Status never reads secret values.
With a login, status also reads the agent job ACL. If its owner is a user, status
warns on stderr that removing that user stops the job under
restrict_workspace_admins. This warning does not change the JSON server reads.
Add-eval and disconnect need an account owner/admin and a Databricks workspace-admin
OAuth login. With several workspaces, pass --host. Add-eval attaches the route and
folder only after checking the workspace admin, reading the folder, and exporting
and validating the YAML manifest. It then grants folder CAN_READ with PATCH and
grants the agent SP SELECT on its cases table. It also grants USE_CATALOG and USE_SCHEMA
when those namespaces differ from the binding. It uses the runner's validation rules
for the manifest fields it reads. It prints one databricks secrets put-acl ... READ
command for each declared scope and starts the recorded agent job. No secret value is
accepted. The server refuses routes with repository or hosted evals.
Before requesting closure, Disconnect runs pending SP-change grants and revokes
for the open binding, including with --force. Refusals warn and stay pending.
Disconnect requests closure and starts the recorded agent job. It waits up to three
minutes for archived or disconnected, then checks and deletes the closed binding's
recorded objects. It polls the binding every two seconds. Before deleting, it
reads the open-binding list once and each other open binding once. It preserves
ids they hold. --force closes at once and runs only that binding's server cleanup steps as
the admin. Jobs, secret scopes, serving endpoints, manageable provider services and
the two working volumes are removed. The results table and experiment remain.
The agent SP remains for an account admin to remove in the account console.
Unmanageable objects retain their cleanup instructions. Successful cleanup is
acknowledged with {done: [{kind, id}]}. Repeating disconnect can resume cleanup
for a closed binding without another disconnect request or agent run.
--reclaim needs --host. It creates a fresh identity-only proof and closes the
holding binding without requiring a role in its BenchRouter account. It reads no
holding binding. It removes only the returned endpoints and manageable provider
services. It cannot use --force or --results-owner.
--results-owner <group> sends results_owner: {type: "group", name: <group>}.
Without the flag, the field is omitted and the results table and experiment stay
with the agent SP. The server rejects string principals with invalid_results_owner.
These commands require the H1/H2 headless server contract.
For local replay tests only, BENCHROUTER_DATABRICKS_API_BASE redirects Databricks
REST calls to a loopback HTTP server. Keep it unset for real workspace setup.
New SP rule-set GET and PUT requests retry permission propagation failures with
up to six tries with 2, 4, 8, 16 and 30 second waits shared across both calls (60 seconds total).
Only 403 or PERMISSION_DENIED failures retry, and only for an SP created in the current run.
Recorded SP reads do not retry.
For tests only, BENCHROUTER_DATABRICKS_TEST_RULE_RETRY_MS (0 to 100) shortens
those waits when the loopback replay override is active.
If workspace discovery does not return the Databricks account id, set
DATABRICKS_ACCOUNT_ID to that id and retry.
Initialize a repository
Run init from the repository that will use BenchRouter:
npx @benchrouter/cli init \
--setup-key br_setup_... \
--route-id product/route \
--name "Route Name" \
--incumbent-model provider/model
Pass --eval-command "pnpm run test:customer" only when an existing customer
test command is used for local capture. Without it, setup emits no
capture_command. The signed CI runtime command is separate.
For a direct-provider incumbent, also pass --provider-id <id> and
--provider-ref <exact-ref>. Pass both or neither.
Pass --base-url-env <NAME> with the environment variable the call site reads
for its LLM base URL, and --api-family <family> with the protocol the call
site speaks: openai_chat_completions, anthropic_messages, or
openai_responses. The family chooses the BenchRouter base URL value, not the
variable's name: a native Anthropic Messages call site gets
https://api.benchrouter.com, because the Anthropic SDK adds /v1/messages,
and the OpenAI families get https://api.benchrouter.com/v1. --api-family is
required for a route that sets --base-url-env or --eval-pack. Repeat either
option once per --route-id, in the same order, or pass it once for all routes.
The CLI sends both as the route's call_site, and setup records them in
.benchrouter/benchrouter.yml. For a route that is already in that file, init
keeps the local entry and writes only the --api-family you pass into its
call_site.api_family (and into a repository-executable eval's api_family).
BenchRouter preserves that exact observed tuple during setup. If the canonical
incumbent cannot serve, init stops and lists only replacements backed by
catalog evidence. The CLI prints the browser approval page. A signed-in GitHub
user with repository write access selects the replacement there. Then rerun the
exact same init command unchanged. The server binds the browser approval to
the setup session, route, exact observed model, provider, provider reference,
and offered options. An unknown identity stops for catalog review. It does not
become a free-form model selection.
Repository-executable evals
If quality depends on the repository's full pipeline, pass one validated JSON
eval pack for each route. Repeat --eval-pack in the same order as --route-id:
benchrouter init ... --eval-pack .benchrouter/contextual-synopsis-eval.json
The file must use this contract:
{
"mode": "repository_executable",
"id": "contextual_synopsis_v1",
"config_path": ".benchrouter/benchrouter.yml",
"workflow": ".github/workflows/benchrouter-evals.yml",
"command": "npm run eval:contextual-synopsis",
"scorer": ".benchrouter/scorer.contextual-synopsis.js",
"result_schema": "benchrouter.executable_result.v1",
"case_refs": ["eval/queries.json"],
"argv": ["node", "eval/contextual-synopsis.mjs"],
"runtime": "node",
"runtime_version": "22.18.0",
"lockfile": "package-lock.json",
"input_refs": ["eval/contextual-synopsis.mjs", "eval/corpus.json"],
"acceptance_refs": ["eval/queries.json", "eval/qrels.json"],
"result_path": ".benchrouter/executable-result.json",
"primary_metric": "recall_at_5",
"timeout_minutes": 60,
"secret_env": ["OPENAI_API_KEY"]
}
The eval pack has no api_family. It inherits the route's --api-family, so
pass the family of the call site the evaluator exercises. The CLI rejects a pack
that declares its own.
The CLI rejects mutable runtime versions, unsafe paths, missing referenced
files, invalid or reserved secret names, fractional timeouts, and timeouts above
90 minutes before it sends the setup request. The eval pack declares no call or
spend limit: BenchRouter bounds each executable run's spend itself, at five
times the route baseline's measured spend on the same eval or $1, whichever is
higher. The CLI rejects max_model_calls, max_cost_usd and
max_cost_per_call_usd as unknown fields. The generated workflow runs argv without a shell. It receives only
the declared customer secrets and the scoped BenchRouter eval contract.
The setup key comes from the signed-in BenchRouter setup page. It is scoped to
one GitHub repository. A successful setup can return one runtime key:
BENCHROUTER_API_KEY. Install that key only in the application host.
BenchRouter Evals does not use a stored GitHub Actions key. The generated
workflow uses GitHub OIDC with id-token: write to get a short-lived eval token.
Do not create BENCHROUTER_EVAL_API_KEY.
By default, init does not save the setup token. Pass --save-token, or approve
the interactive prompt, to keep repo-scoped read access on the current computer.
For non-interactive use, set BENCHROUTER_TOKEN instead.
npx @benchrouter/cli init ... --save-token
The generated files include .benchrouter/SETUP_README.md. Read that file
before changing the call site, eval cases, or scorer.
Signed runtime commands
Run these commands from the repository with the generated kit:
benchrouter capture
benchrouter calibrate
Both commands execute .benchrouter/bootstrap.mjs. The bootstrap verifies the
signed release before it starts. Capture starts a local proxy. Run the existing
application test command separately against that proxy. Calibrate checks local
evidence. Neither command proves production readiness.
The generated GitHub Action uses bootstrap.mjs run. The benchrouter run
entrypoint forwards to the same bootstrap and requires GitHub Actions OIDC.
The workflow uses one Node 24 job with a 120-minute ceiling. It grants evaluator
secrets and customer setup only to server-dispatched work. The trust file carries
current and next signing keys, with digest pins empty by default.
Commands
benchrouter init --help
benchrouter upgrade --help
benchrouter doctor
benchrouter models [--filter text] [--json]
benchrouter status [--json]
benchrouter frontier <route-key> [--json]
benchrouter failures <route-key> [model] [--json]
benchrouter explain <model> [--route <route-key>] [--json]
benchrouter login [--name text] [--expires-in-days 90]
benchrouter account show [--json]
benchrouter account token save --account-token br_ctrl_...
benchrouter billing show|activity|statement [--json]
benchrouter billing top-up --amount 25 [--yes]
benchrouter billing auto-reload setup --threshold <usd> --amount <usd> | disable
benchrouter keys list|create|disable|enable|revoke
benchrouter repos list|refresh
benchrouter setup status [--repo owner/repo]
benchrouter setup create --repository-id <id> --installation-id <id> [--intent initial|new_route]
benchrouter setup session show <session-id>
benchrouter setup approve <approval-id|approval-link> [--model <model-id>] [--yes]
benchrouter setup upgrade-token --route-id <id> [--repo owner/repo]
benchrouter routes list|show|catalog|activity|startup-failures|create|pin|archive|unarchive
benchrouter evals list|run
benchrouter evals status <route-key> [--pr <number>]
benchrouter evals results <route-key>
benchrouter evals model <route-key> <model-id> [--pr <number>]
benchrouter evals case-output <route-key> --run <model-run-id> --case <case-id>
benchrouter evals overview
benchrouter evals draft --name <text> --cases <cases.jsonl> --scorer <scorer.json>
benchrouter evals approve <draft-id> --card <card-hash> [--yes]
benchrouter evals run <route-key> --full-refresh
benchrouter evals retry --result-set <id> | --route <route-key>
benchrouter evals harness-retry <route-key> [--model <id>]
benchrouter evals refresh-preview <route-key> <result-set-id> [--model <id>]
benchrouter blocks list [--repo owner/repo]
benchrouter blocks unblock --repo owner/repo [--route <route-key>] | --route <route-key> | --all-blocked
benchrouter public status|frontier [--json]
benchrouter baseline set <route-key> --result-set <id> --model <id> [--yes]
benchrouter proposals list|approve|reject [--admin-token bradm_...]
benchrouter admin providers|catalog|keys|token [--admin-token bradm_...]
benchrouter admin catalog show|activity|model-maps|refresh-report|drain-outbox|rebuild
benchrouter admin catalog observations [add]|mappings [list|resolve|ignore]
benchrouter admin keys list|revoke [--admin-token bradm_...]
Run benchrouter login to authorize the CLI in your signed-in browser. The CLI
keeps the verifier locally, waits for approval, receives the new br_ctrl_
token directly, saves it in private local config, and proves the selected
account. The browser URL contains only an opaque transaction ID. The raw token
never appears in the URL, browser storage, terminal output, or logs.
Account commands authenticate with a br_ctrl_ account token
(--account-token, then BENCHROUTER_ACCOUNT_TOKEN, then private local
config). Proposal/admin commands use a bradm_ admin token
(--admin-token, then BENCHROUTER_ADMIN_TOKEN, then private local config).
Repo-read commands keep using the setup/read token. Tokens are never
auto-substituted across scopes. Runtime API keys never authorize control-plane
commands.
Mutations print an exact action summary and prompt unless --yes. JSON mode
never prompts and requires --yes. Billing top-up prints a checkout URL; it
does not open a browser. billing show reads billing fields from
GET /v1/dashboard/summary.
Create a pinned route in a new group or an existing group with no repository:
benchrouter routes create --name Triage --model anthropic/claude-haiku-4.5 --group Support --group-slug support --route-slug triage --yes
benchrouter routes create --name Triage --model anthropic/claude-haiku-4.5 --group-id <id> --yes
Pass exactly one of --group <name> or --group-id <id>. --group-slug <slug>
requires --group. Slugs default to the names. Creating a new group requires an
account owner. The request sends group: {name, slug?} or group_id.
The response keeps the internal product field and reports group_created.
To create a runtime key for the group, use benchrouter keys create --product-id <id>.
That endpoint still requires product_id.
status shows each route, incumbent, current best model, production wiring state,
latest eval state, and production result-set ID. Use status --json when an agent
or script must prove that the route has received a production call and identify
the exact evidence used in production.
frontier shows the incumbent, best model, and ranked alternatives.
failures shows failed cases from the latest model run. Pass a model ID to
select the latest run for that model.
explain calls the server model-explanation endpoint and states whether a model
is the incumbent, best pick, an eligible alternative, or outside the eligible
frontier. Pass --route when a repository has more than one route.
keys revoke <key-id> calls POST /v1/dashboard/api-keys/:keyId/revoke and
prints non-secret key metadata. Revocation is immediate: any application still
using that key stops authenticating.
setup create starts a setup session and prints a one-time setup code plus the
server-authored init command. setup upgrade-token mints a single-use
br_upgrade_ token and prints it once. Both secrets are printed once and never
saved; pass them to init --setup-key and upgrade --upgrade-token. Read
repository_id and installation_id from repos list.
blocks list prints one line per block that stops evaluations: its kind, scope,
time, whether Unblock clears it, and how to clear it. blocks unblock (account
owners) clears BenchRouter's own breakers (startup_failing,
snapshot_refresh_failing, harness_failing) and re-checks workflow_missing
and installation_revoked against GitHub. Queued evaluations restart through
the normal dispatcher; Unblock creates no evaluation and reruns no model.
evals retry reruns the failed models of one production evaluation, or wakes it
when it is still open. Retry never clears a block, so a blocked route is
refused until you unblock it.
evals refresh-preview re-dispatches an open-PR preview result set. The result
set must be a PR preview; pass --model to refresh one model only.
admin catalog refresh-report is POST, not GET. It is report-only: the
server fetches upstream state and performs no writes.
admin catalog drain-outbox [--limit 1..25] publishes a bounded amount of
durable catalog work and reports both completed work and the remaining backlog.
It is an admin mutation, so it requires a bradm_ token and confirmation (or
--yes). A br_ctrl_ account token cannot authorize it.
admin catalog observations add records a manual notice; pass structured fields
through --payload-json, which must parse to a JSON object.
admin keys list and admin keys revoke <key-id> accept a bradm_ bearer and
print non-secret metadata only. Minting an admin key requires a browser GitHub
admin session, so the CLI has no mint command. Use admin token save /
account token save for already-minted tokens. Save commands never print the
secret. Use account token save only as a recovery path. Use benchrouter login for normal account authorization.
All read commands accept --json for scripts and agents.
Credentials and configuration
Read commands resolve credentials in this order:
--token br_setup_...BENCHROUTER_TOKEN- the saved token for the detected or specified repository
Saved credentials are isolated by repository (and account/admin files are
owner-only mode 0600). Set BENCHROUTER_CONFIG_DIR to move the entire
configuration root. Tests and automation should always set this variable to a
temporary directory.
The CLI does not write runtime keys to disk. It never saves a setup token unless the user approves the write.
Doctor
doctor checks the generated files, package-script wiring, and the GitHub OIDC
workflow. Activation mode also checks the selected runtime call site. For isolated replay, it also checks
runnable case arrays and scorer syntax. For repository_executable, it checks
that the declared lockfile, case, input, and acceptance refs exist. It does not
apply isolated-replay JSON shapes or scorer execution to repository files.
The runtime checklist uses the route's call_site.base_url_env and
call_site.api_family. It prints https://api.benchrouter.com/v1 for the OpenAI
families and https://api.benchrouter.com for anthropic_messages, because the
Anthropic SDK adds its own /v1/messages path. A route without
call_site.api_family gets no value; the checklist names the missing field.
doctor defaults to evaluation checks. It does not require production wiring or a runtime key and never implicitly makes a provider call. Use --phase activation --route-id product/route to check one selected call site.
Add --live-chat-completions only to opt into a billable synthetic Chat Completions diagnostic. An inherited BENCHROUTER_API_KEY alone does not opt in. This diagnostic does not verify the application request contract; use an authorized application smoke for that. Native Anthropic and Responses calls need their own application smoke.
It labels each external check as passed, failed, or skipped. A doctor pass only
means that every check that ran passed. It does not certify evaluation quality
or production readiness. Local calibration does not provide that certificate.
Use benchrouter setup status --repo owner/repo with an account token to read
the server's separate route-registration, evaluation, production-eligibility,
and observed-serving facts. Evaluation completion is not a production
eligibility gate. Observed serving is historical delivery evidence, not a
current health check.
benchrouter doctor --repo owner/repo --skip-github-workflow
benchrouter doctor --phase activation --route-id product/route --repo owner/repo --check-default-branch
Use --skip-github-workflow when gh is unavailable or the workflow does not
exist on the default branch yet.
Multiple routes
Repeat --route-id, --name, and --incumbent-model in the same order:
benchrouter init --setup-key br_setup_... \
--route-id product/route-a --name "Route A" --incumbent-model provider/model-a \
--eval-pack eval/route-a-pack.json \
--route-id product/route-b --name "Route B" --incumbent-model provider/model-b \
--eval-pack eval/route-b-pack.json
Each runtime call site uses its stable route ID as the OpenAI-compatible model
value. Do not create one global model variable for a repository with several
routes. If any route uses --eval-pack, every route in that init command must
have one positionally matching file.
Upgrade
upgrade previews the bootstrap, trust file, workflow, and README, asks for
confirmation, then applies them. It preserves .benchrouter/benchrouter.yml
byte-for-byte. That YAML is the single route declaration. Upgrade validates
each file hash before it writes any file. It preserves customer setup steps
between Setup Node.js and Run BenchRouter. Each customer step must use
if: github.event_name == 'workflow_dispatch'. It keeps the eval job's
runs-on, so a job moved to self-hosted runners stays there; doctor accepts
any runs-on label, label list, or group/labels object. It never replaces cases,
scorers, calibration fixtures, setup guides, or app files. Old generated-engine
installations must be removed and re-onboarded; they have no compatibility path.
benchrouter upgrade \
--upgrade-token br_upgrade_... \
--repo owner/repo \
--route-id product/route
Use --dry-run to preview a single-use upgrade token without applying it. Use
--yes only when an interactive confirmation is not possible.
Model IDs
models prints the current BenchRouter catalog. A route incumbent can be an
exact OpenRouter model that is not an automatic candidate. If BenchRouter cannot
resolve the incumbent identity, stop for catalog review. Do not substitute a
model or remove the observed provider metadata. If a resolved incumbent cannot
serve, open the server-provided browser approval page. Then rerun the exact same
init command unchanged.
init prepares evaluation files and refreshes the generated BenchRouter workflow so added routes receive their declared triggers and executable secret mappings. It preserves production call sites and existing env examples. Add the runtime key and endpoint configuration in the later activation patch.