@appforge-ci/admin-cli

appforge-admin — the platform-operator CLI for fleet management (enroll, list, drain, resume, reboot, run maintenance scripts on Mac minis). This is a separate tool from appforge (packages/cli), which is for AppForge's customers and has no fleet commands at all — see docs/self-hosting.md's "Fleet management" section for why they're kept apart. Config lives in ~/.appforge-admin/config.json, independent of ~/.appforge/config.json.

For the full day-to-day runbook (reading fleet status, troubleshooting a NotReady/offline mini, rotating tokens, decommissioning) once this is installed, see docs/fleet-management.md.

Install

On a machine that already has this repo cloned (the common case — your own laptop, or a Mac mini you git cloned onto): build it and put it on PATH with npm link:

pnpm install                              # once, from the repo root
pnpm --filter @appforge-ci/admin-cli build
cd packages/admin-cli && npm link         # symlinks `appforge-admin` onto PATH

npm link needs to be re-run only if the symlink itself ever gets removed (e.g. npm uninstall -g @appforge-ci/admin-cli) — a plain rebuild (pnpm --filter @appforge-ci/admin-cli build) is enough to pick up code changes afterwards, since the link points at dist/ in place.

From npm (the way a Mac that has no checkout gets it): .github/workflows/publish.yml publishes this package on every release tag (git tag vX.Y.Z && git push origin vX.Y.Z, job publish-clis), so npm install -g @appforge-ci/admin-cli@latest installs it and re-running that is how you update it. The agent's self-update does not touch this install on a release-mode Mac (#852): after a Mac's agent moves to a release that added a Mac-side command (install-vm-user, ...), update the CLI with the same command, or the command fails there with unknown command. A repo-checkout Mac whose agent self-updates also rebuilds this CLI itself, but only when its checkout already built it.

Usage

appforge-admin login --api-url <api-url> --admin-api-url <admin-api-url>   # once; add --web to approve in a browser instead of typing your password here
appforge-admin logout                                  # remove the login from THIS machine (the file and leftover copies) AND revoke its token on the server — see below
appforge-admin agents list
appforge-admin agents add-mac --name "my-mac-mini"    # run ON the mini you're enrolling — see docs/self-hosting.md; it ends the login there itself (--keep-login keeps it)
appforge-admin agents add-mac --force                 # re-enroll THIS Mac (rotate token / rename) — same fleet row, never a duplicate
appforge-admin agents decommission <agentId>          # permanently remove a mini (retired machine, or a stale duplicate row)
appforge-admin agents drain <agentId>
appforge-admin agents resume <agentId>
appforge-admin agents reboot <agentId>
appforge-admin agents restart <agentId>       # just the agent process (e.g. after a manual upgrade) — no reboot needed
appforge-admin agents cancel-build <agentId> <buildId>  # kill one wedged build, other concurrent builds on the same mini untouched
appforge-admin agents allow-remote-control <agentId> --on|--off  # authorize (or revoke) the VNC "Connect" button's mouse/keyboard control — off by default for every agent
appforge-admin agents allow-vpn <agentId>      # approve a Mac for site-to-site VPN duty (platform admin) — off by default; also deny-vpn <agentId> to revoke
appforge-admin agents run-script <agentId> <scriptName>
appforge-admin agents commands <agentId> [--limit 20]  # remote-command history + who issued each one — see docs/fleet-management.md
appforge-admin agents update <agentId> [--to <version>]   # roll one mini onto a new agent build (OTA) — see docs/fleet-management.md
appforge-admin agents update --all                        # every enrolled mini, one at a time
appforge-admin agents install-vm-helper [--no-gateway] [--allow-local-gateway]  # per-build VMs: the root-owned VM helper + sudoers line, and the VM gateway binary this package carries (root-owned, see below)
appforge-admin agents bootstrap [--full]              # install missing toolchains on THIS Mac
appforge-admin agents enroll                           # advanced: just a token, no local enrollment — see add-mac instead
appforge-admin agents label <agentId> canary m4        # set a Mac's operator labels (replaces them; selectors match label:canary); --clear removes all
appforge-admin fleet steps list                        # setup-step catalog: per step how many Macs applied / missing / failed / unknown, and which restart something (drain first)
appforge-admin fleet steps status [--step <id>]        # drift per Mac; a Mac that has not reported is shown as "not reported", not as drift
appforge-admin fleet steps apply <stepId> --selector <text> [--canary N] [--batch N] [--max-failures 10%|N] [--timeout MIN] [--force] [--dry-run] [--wait] [--yes]
appforge-admin fleet steps rollouts [--limit N]        # recent rollouts
appforge-admin fleet steps rollout <id>                # one rollout: policy, who/when, halt reason, per-Mac result
appforge-admin fleet steps cancel <id>                 # running Macs finish, the rest are skipped

Rolling a setup step out to many Macs

fleet steps apply is the replacement for ssh -t host 'sudo ...' once per Mac (design: docs/fleet-steps.md). The selector is required and never defaults to the whole fleet: all, ids:a,b, name:mini-*, label:canary, percent:25, missing:<step>, failed:<step>, joined with + (AND). Try it first with --dry-run: it prints "N Macs in M batches (canary first), K already applied" and a per-Mac table (batch, current step state, skip reason) and creates nothing. A real run prints the same plan and asks for confirmation unless --yes, then prints the rollout id; the rollout runs on the server whether or not the CLI stays open. With --wait the CLI follows it, printing each Mac's progress, and exits non-zero if the rollout halted, was canceled or any Mac failed (the halt reason and the failed Macs' result text are printed). Ctrl-C while waiting only stops watching; fleet steps rollout <id> shows it later and fleet steps cancel <id> stops it. The server's own messages for a bad step id, selector or policy are printed as they are. A step that restarts something drains the Mac first.

--api-url/--admin-api-url only need passing on that first login — they're saved to ~/.appforge-admin/config.json and reused after that. Only accounts listed in the admin-api Worker's PLATFORM_ADMIN_EMAILS secret can actually log in here; see docs/self-hosting.md.

The password-based login (not --web) prompts interactively for a 2FA code if the account has it enabled and neither --totp-code nor --recovery-code was passed — see docs/security.md's "2FA (TOTP)" note. login --web's device-authorization flow has no password step at all, so it's unaffected either way.

Never leave the login on a Mac that builds

login stores a long-lived afk_ API token in ~/.appforge-admin/config.json. A platform admin's token drives the whole fleet (drain, reboot, update, VNC, enrolment), and a Mac that builds for tenants runs their standard-tier builds as the same user, so any of them can read that file. Keep the login on your own machine. If you must log in on a build Mac (for agents add-mac), the login ends by itself when add-mac finishes; for anything else, run appforge-admin logout the moment you are done:

• appforge-admin logout deletes config.json and any config.json.* copy next to it (the temp file of a killed save, a .bak), prints what it removed, leaves everything else in the directory alone (a directory is never entered), and is safe to run when nothing is there. A config.json that is a symlink is unlinked, not followed: the output names the file it pointed to, which stays.
• It revokes the token on the server too (DELETE /auth/token on --api-url, the auth API; issue #952): the token stops working at once, so a copy someone already read is dead. The files are read first and removed before any network call, so the Mac is clean even if the API is unreachable; every saved copy's token is revoked, each against the API URL saved beside it. If the revoke fails (offline, a control plane that predates the route) logout still removes the login, exits 0, says it could not revoke, and prints the manual follow-up to run from a trusted machine: appforge tokens list (the token is labelled admin-cli:<email> after a password login, admin-cli:device-flow after login --web; match it by its created time), then appforge tokens revoke <id>, and log in again only there. Deleting the file by hand revokes nothing. The token is never printed.
• login prints a warning when builds run as the account you logged in with (the agent's config or LaunchAgent plist is in your home, or the boot-time agent runs as you); a laptop with no agent gets nothing, and neither does another account on a Mac whose agent runs as someone else. agents add-mac always ENDS the login when it finishes (also when the service install failed, since the Mac is enrolled by then), because it has just made this account a build account: it revokes the token and removes the file, the same as logout. agents add-mac --keep-login keeps the login to run more appforge-admin commands from that Mac and prints a one-line reminder instead; run logout when done. appforge-agent doctor fails (cli-credentials-off-build-mac) while a login is stored on a Mac with an agent. The warnings and the check refuse nothing. When a revoke has to be done by hand, appforge tokens list is newest first, 50 to a page: add --offset 50 if the old login's token is not on the first. To avoid ever logging in on the Mac, enrol it with a one-time token minted on your own machine (appforge-admin agents enroll, then appforge-agent enroll on the Mac; infra/mac-mini-setup/provision-remote.sh does this over ssh): docs/fleet-management.md, "Enrolling a Mac without leaving a login on it".

Details and limits: docs/security.md, "A CLI login never stays on a build Mac". The token self-revoke (issue #952) is covered by unit and route tests; not yet run against the deployed control plane (UNVERIFIED until a dated run is recorded there).

The VM gateway binary

appforge-admin agents install-vm-helper also installs the VM gateway ("softnet", docs/vm-gateway.md), the userspace network that will carry the site-to-site VPN into a build VM, at /etc/appforge/bin/softnet (root-owned, 0555) with its hash recorded in /etc/appforge/softnet.manifest.json. The binary ships inside this package, next to a manifest: dist/softnet and dist/softnet.manifest.json (sha256, size, commit, version, Go toolchain, and source: ci or local). install-vm-helper verifies the binary against the manifest (size, SHA-256, arm64 Mach-O header) before it changes anything and refuses to continue on a mismatch; it also refuses, changing nothing, a package whose manifest says local (a binary somebody built or copied in, the lab build included) unless --allow-local-gateway is given, and a manifest that says ci for the lab binary always; --no-gateway leaves the gateway out. It is not part of the agent's signed OTA bundle on purpose (that bundle is unpacked where the agent's account can write; see docs/security.md).

The published package always carries the release workflow's build. .github/workflows/publish.yml cross-compiles it from the tag's commit (reproducibly: -trimpath, the go.mod toolchain, a second build with an empty cache must give the same bytes) and runs node scripts/vm-gateway.mjs check --require-ci --version V --commit SHA before pnpm publish, which refuses a package whose manifest is not the CI build of that commit.

Local development / a checkout (what works today, until the npm publish is configured). pnpm --filter @appforge-ci/admin-cli build runs tsc and then scripts/vm-gateway.mjs stage, which leaves a dist/softnet that already has a matching manifest alone, gives one that was copied in (or built with make vm-gateway in packages/vpn-proxy) a local manifest, and does nothing, without error, when there is no binary. To build one with the release's own flags:

pnpm --filter @appforge-ci/admin-cli build
pnpm --filter @appforge-ci/admin-cli gateway:build     # needs Go; writes dist/softnet and dist/softnet.manifest.json (source: local, version dev)
node packages/admin-cli/scripts/vm-gateway.mjs check   # ok (softnet dev, commit ..., local build, sha256 ...)
appforge-admin agents install-vm-helper --allow-local-gateway   # on a lab Mac, as the agent's account, without sudo (a released package needs no flag)

gateway:build sets GOFLAGS itself, so a developer's -tags=vmgwlab cannot produce the lab binary by accident; the lab build (make vm-gateway-lab) is only for the benchmark in docs/manual-vm-validation.md section 8.7 and must never be installed on a Mac that builds for anyone. A local build needs appforge-admin agents install-vm-helper --allow-local-gateway (a lab Mac only), and the Mac then runs it for run --gateway but does not advertise the VM VPN feature for it: only the release workflow's build, at or above the agent's minimum version, earns that. pnpm publish of this package runs check --require-ci first (prepublishOnly), so a hand-run publish of a local build fails. The full install-and-verify recipe for a real Mac is docs/manual-vm-validation.md section 9.