@agent-forge/plugin-desktop-macos-input

Bounded macOS desktop input capability plugin for Agent Forge, structured as the macOS counterpart of plugins/desktop-windows-input. It registers desktop_execute — focus, click, type, key, scroll — backed by the macOS Accessibility tree through a JXA driver (osascript -l JavaScript). A readonly host mode registers no input tools at all, and the shipped facade deliberately exposes no replay method, so desktop_replay never registers out of the box.

Action surface: click accepts observed control identities (AXPress with a CGEvent coordinate-click fallback) and — click-only — explicit coordinates targets (integer, non-negative screen coordinates); key uses a minimal SendKeys-aligned grammar (+ shift, ^ control, % option, {ENTER}-style special keys) translated to System Events keystroke/keyCode; scroll posts CGEvent scroll-wheel events. High-risk actions (uac, password, admin, payment, external-submit) stay behind the shared requiresDesktopUserConfirmation fail-closed gate and always return needs_user. Every action re-observes so results carry fresh state.

On non-macOS platforms importing the package does not throw; every tool reports an explicit unsupported result. A missing Accessibility (assistive access) grant is reported fail-closed as the probe reason accessibility-permission-required.

Install

# from a local checkout
agent-forge plugin install ./plugins/desktop-macos-input

# from npm
agent-forge plugin install @ai-agent-forge/plugin-desktop-macos-input

The package ships two manifests: agent-forge.json (ecosystem manifest for the deterministic plugin resolver, platform.os: ["macos"]) and plugin.json (host-load manifest consumed by loadCapabilityPluginFromManifest, entry ./dist/entry.js).

Library surface

import { DesktopMacosInputAdapter, MacosAccessibilityDriver } from "@agent-forge/plugin-desktop-macos-input";

• DesktopMacosInputAdapter — production input facade (no replay method).
• MacosAccessibilityDriver(options?) — JXA driver; supports command/timeoutMs/spawnSync/platform injection for brokers and tests.
• createDesktopInputCapabilityFactoryV1(runtime) — registers desktop_execute (plus desktop_replay only when the injected runtime exposes a replay method) on a CapabilityAPI.
• Default export of dist/entry.js — the precomposed factory used by external plugin loading.

The session/observation/projection contract and the confirmation gate are deliberate inlined copies shared with @agent-forge/plugin-desktop-macos-observe so both packages stay self-contained.

Real-host status

The driver and adapter are contract-tested with an injected fake spawnSync (canned JSON, no real desktop). Real darwin behavior — the JXA enumeration and action scripts, Accessibility permission classification, keystroke/keyCode delivery and CGEvent clicks — is verified only by the explicitly gated test (AGENT_FORGE_MACOS_AX_INPUT_REAL_TEST=1 on macOS, scroll-only to stay reversible) and has not been validated on a real macOS host as of this change.

Development

npm install --ignore-scripts        # repo root, creates the workspace symlink
npm run build                       # plugins/desktop-macos-input (tsgo)
npm test                            # vitest; the entry fidelity test requires `npm run build` first