@alwith/module-voice

Private release candidate. Streaming speech recognition for VUI and Doubao, with optional React and editor adapters. No credentials are included.

Public entry points

React and React DOM are optional peer dependencies, required when consuming /react. The package leaves React, React DOM, and the JSX runtime external; consumers supply their own copies. React output uses the production JSX transform. The other entry points do not require React.

Recording and editor interaction

VoiceController owns one recognition operation. Call start(options) with a new recording ID, a credential profile, hotwords, and the provider smoothing option. State, partial text, final text, and errors arrive through callbacks. finish() flushes capture and waits for the operation's final result; failures reject. cancel() interrupts the operation, and dispose() waits for its cleanup. Release the capture permit only after the actual microphone has stopped.

VoiceInteraction adds editor transactions and submission behavior around sessions created by InteractionHost.createSession. The host supplies the draft, admission checks, settings and credential availability, configuration UI, a synchronous edit commit function, submission, turn observation, and user-visible error/no-speech reporting. Configuration UI failures are reported through onError. Settings are loaded for a new interaction and reused across its handsfree rounds.

The three modes are:

• dictation: update the selected draft range; leave sending to the user.
• send: finish on the configured silence/duration limit and submit through the host.
• handsfree: submit, await acceptance and the corresponding turn completion, then record again.

Submission returns { status: "accepted" } or { status: "blocked", reason }; failures reject. The host must install turn observation before submission and correlate completion to the accepted prompt. Acceptance alone, or a busy-state transition, is not completion. The host must reject that observation on approval waits, disconnection, failed turns, or an intervening prompt. Enable handsfree only when this correlation is available.

Automatic send modes refuse an existing nonempty text draft. Empty final text restores the original selection and stops automatic listening. Recognition text is a full replacement snapshot, not a delta. Normal socket closure never substitutes for a final result.

Every user or external draft edit must advance its revision, including same-text writes. DraftPort.compareAndSet must check target identity and revision at the actual write. Cancellation restores only a still-owned draft; failure preserves the displayed preview. Use cancel(false) for external edits and cancel(true) for an explicit user rollback. A successful submission must clear only its submitted draft revision and must not be treated as an external edit that cancels the handsfree cycle. Dispose interactions on target replacement or unmount.

React controls and settings slots

VoiceControls receives host (the interaction host without createSession and commit), transport, a capture factory, enabled, and a stable cancelRef. It creates the controller and silence detector and commits editor changes with React DOM flushSync. Supply the Button component, localized labels, and icons; the package does not impose an application design system. Its cancelRef callback preserves external edits. Keep the draft identity, transport, and cancel ref stable for one editor target; replacing them disposes the interaction.

VoiceSettingsForm receives a VoiceSettingsHost with load, save, profiles, saveCredential, and deleteCredential. Supply Group, Row, Input, Textarea, Button, Switch, and Select through VoiceSettingsSlots, together with text, onError, and onSaved. The Select slot receives a label, value, option list, and onChange; it adapts the host's select implementation. Credential values are entered for saving, never loaded back into the form. Device retries refresh devices independently of unsaved settings and credential input.

defaultPreferences() creates preferences; parsePreferences(unknown) validates persisted input; recordingSettings(preferences, cwd) merges global and exact-project-path hotwords, splitting commas/newlines, trimming, and deduplicating them. Persistence and project identity remain host responsibilities. Smoothing is a Doubao option.

Devices and silence limits

Capture uses mono 16 kHz signed little-endian PCM and rejects unsupported sample rates. An empty deviceId selects the system default; an explicit ID requests that exact device. watchInputDevices(media, changed, failed) enumerates audio inputs, listens for device changes, ignores stale responses, and returns an unsubscribe function. Enumeration does not request microphone permission or open a separate capture stream; device labels and availability depend on browser permission.

Automatic modes use RMS energy over PCM frames. SilenceDetector reports its stop decision once. This is an energy threshold detector, not a speech recognition model.

All values must be finite. Preferences without a silence configuration receive the documented defaults. Once speech exceeds the threshold, trailing silence can finish the round; initial silence and maximum duration provide separate stopping conditions. These limits apply to automatic modes. Native recognition deadlines remain independent.

The frontend queue includes in-flight writes in its byte budget. Native audio is bounded to ten 200 ms frames. Overflow fails explicitly rather than dropping intermediate audio. Cancellation does not wait for queued audio to drain.

Native events and host lifecycle

Recognition and providers live in the private native voice library, shipped in this package under native/<os>-<cpu>/. Tauri hosts use the public plugin alwith-voice from crates.io:

[dependencies]
alwith-voice = "0.6"

.plugin(alwith_voice::init(credentials, Config { development_directory }))

credentials implements alwith_voice::Credentials: the host's credential store, which runs the plugin's step while holding its own lock so a credential change is ordered against a recording start. The plugin owns document identities (it detaches a webview's recordings on navigation and window destruction), event routing to channels, and the commands start, push, control, profiles, saveCredential and deleteCredential. Grant alwith-voice:default (recording and profile availability) to capture windows and alwith-voice:credentials to the settings window. Call alwith_voice::shutdown(app) in the application's ordered exit sequence; call cancel_provider(app, provider) when the host changes a credential outside the plugin's commands. Typed bindings are exported from @alwith/module-voice/commands; createTauriTransport addresses the plugin commands. TLS uses Rustls AWS-LC inside the library.

Events carry a recording ID and, in current native output, a service-assigned generation and increasing per-recording sequence. Partial coalescing can leave sequence gaps. The controller filters mismatched generations and duplicate/older events; the contract keeps metadata optional for existing transports. Native events also carry the bound owner for host routing. Do not infer ownership from the selected chat or a WebView-supplied window label.

Failed events expose a sanitized message and structured VoiceFailure fields: stage, category, code, and nullable requestId. VoiceError.detail preserves this metadata when present. Numeric provider error codes are retained; raw transport/provider error text is not exposed through the native event message. Hosts decide how to localize and display failures. Capture and admission errors can also be ordinary JavaScript errors without native details.

The host resolves credentials without returning stored keys to the WebView, binds ownership to its authenticated window/document identity, and acknowledges actual microphone release. Detach only when the old document can no longer capture; deny new capture during document replacement. Call shutdown before application exit and join the event reader before completing host teardown. The host-only cancelProvider command terminates that provider's active sessions while retaining capture ownership until release acknowledgment.

ALwith U credential integration

U stores provider credentials in macOS Keychain or Windows Credential Manager through its native credential adapter. Other hosts supply their own secure credential resolver; OS credential storage is not part of the browser package. Unsupported secure-storage platforms fail explicitly.

On first access, U migrates a matching credential from its legacy voice-credentials.json only if no secure record exists. Migration must successfully write secure storage before returning the credential. The legacy file is preserved and is not automatically deleted. A locked store or failed write is reported; the adapter does not fall back to using the plaintext key.

Revocation writes an empty secure record as a durable marker, preventing the retained legacy file from reimporting the removed key. Replacing or revoking a credential also cancels that provider's active native sessions. Credential access and native start acceptance share a lock so a queued start cannot bypass revocation. Stored keys are never returned by the profile-availability API.

Build and verification boundaries

From this package directory:

bun run build
bun test src
bun run verify
bun pm pack
cargo test --manifest-path rust/Cargo.toml   # UPDATE_BINDINGS=1 regenerates src/generated/commands.ts
cargo test -p alwith-voice-impl --manifest-path ../../Cargo.toml

The native library is built per platform and assembled into native/ at release. Run Rust tests on each supported host platform before release. The single THIRD_PARTY_LICENSES.txt contains the native library's full third-party license, copyright, and notice texts and must accompany the application together with this package's own LICENSE. There is no change to the package licence.

Automated coverage includes PCM tails, cancellation during capture setup, queue bounds, draft ownership, controller finalization, handsfree acceptance/completion, stale callbacks, preferences, silence detection, and device-list lifecycle. Native tests cover protocol frames, local WebSocket peers, event metadata/sanitization, provider cancellation, and shutdown. U has integration tests for shared controls/settings, document generations, and credential migration/revocation using a fake secret backend. The shared-crate integration test creates and shuts down a service without any library file. Local peers and fake credentials do not establish compatibility with the live services or actual OS credential dialogs.

The prepack hook rebuilds JavaScript and checks that the plugin stays an adapter, crate/package version alignment, that rust/ is not packed, the framework-independent core entry, and production JSX output.

Rebuild and test the exact installed artifacts after changes. In particular, check production React output uses react/jsx-runtime; development renders alone do not validate it. Test counts from another revision are not evidence for this candidate.

Real VUI/Doubao recognition, OS credential prompts, microphone permission/device behavior, Windows runtime behavior, and signed application acceptance remain separate verification requirements. No automatic provider fallback, background retry, transcript persistence, or audio logging is enabled. Complete live-provider and signed-platform verification before production rollout.

Doubao resource selection

VoicePreferences.doubaoResourceId configures the X-Api-Resource-Id header. The default is volc.seedasr.sauc.duration; older saved preferences gain this default when loaded. The Doubao settings form accepts another resource enabled for the account, such as volc.seedasr.sauc.concurrent. It is ordinary configuration, not a credential. Missing or blank values select the default resource. Invalid header characters are rejected.

The host passes the value as StartOptions.resourceId. Omission preserves the default for existing hosts. VUI does not send this header. Settings are read at the start of each recording, including subsequent handsfree rounds; an active recording keeps its original resource. Resource errors are reported without automatic resource switching.

Keyboard gesture

Hosts can enable VoiceControls.optionTap and provide a localized shortcutHint. In the active chat window, two clean Option/Alt taps within 500 ms start recording; one tap finishes recording. One tap during connection or the handsfree reply wait cancels that interaction. Finalization ignores taps. Modified chords, long holds, key repeat, IME composition, pointer interaction, and loss of window focus reset the gesture. Open menus/dialogs prevent starting but do not block finishing or cancelling an active interaction. U enables this gesture, matching the Desktop voice branches. No OS-global shortcut is registered.

Headless React interaction

useVoiceInteraction from @alwith/module-voice/react provides complete voice interaction without rendering UI. VoiceControls uses the same hook internally. Capture, streaming draft updates, silence detection, submission, handsfree cycles, optional Option/Alt gestures, and cleanup remain shared.

UseVoiceInteractionOptions accepts host, transport, capture, enabled, optional optionTap, and optional stable cancelRef. The host contract matches VoiceControls.host. Keep draft, transport, and cancel ref identities stable for an editor target; changing them disposes the old interaction. Disabling cancels active interaction and prevents new recording. The cancel ref preserves current draft text.

The result exposes state (phase, waiting) and stable actions:

• start(): Promise<void> starts a recording.
• finish(): Promise<void> finishes and awaits the provider's final result.
• toggle(): Promise<void> starts, finishes recording, or cancels connection/reply waiting; finalization ignores toggles.
• cancel(restore: boolean): void cancels; true restores the original owned draft, false preserves current text.

Operational errors are reported through host.onError. Handle action promise rejections as shown below; actions called after unmount reject/throw.

import { useVoiceInteraction, type UseVoiceInteractionOptions } from "@alwith/module-voice/react"
import type { ReactNode } from "react"

function CustomVoicePanel(options: UseVoiceInteractionOptions): ReactNode {
  const { state, toggle, cancel } = useVoiceInteraction(options)
  return (
    <section aria-label="Voice input">
      <output aria-live="polite">{state.waiting ? "Waiting for reply" : state.phase}</output>
      <button
        disabled={!options.enabled || state.phase === "finishing"}
        onClick={() => { void toggle().catch(options.host.onError) }}>
        {state.waiting || state.phase === "starting" ? "Cancel" : state.phase === "recording" ? "Finish" : "Start"}
      </button>
      <button onClick={() => cancel(true)}>Discard recording</button>
    </section>
  )
}

Mount one interaction hook per active composer. Do not mount it alongside VoiceControls for the same draft: that component already owns an instance. The host owns layout, styling, labels, icons, and animations. The lower-level useVoice hook remains available for single sessions without chat interaction orchestration.

Consolidated third-party licenses

Run bun run licenses after changing Rust dependencies. It collects the locked macOS ARM64 and Windows x64 dependency graphs and writes THIRD_PARTY_LICENSES.txt, including component identity, version, source, and complete license/notice texts without deduplication. bun run licenses:check regenerates the expected content in memory and fails on differences; prepack runs this check.

license-evidence.json is build input for three upstream crate archives that omit license files. It retains exact texts and pinned upstream URLs, is not included in the npm package, and requires review when those versions change. Hosts using different dependency versions or additional targets must include the notices applicable to their own resolved build.

The host stages only LICENSE and THIRD_PARTY_LICENSES.txt for voice. Other modules' license files are independent.

Codex realtime calls

useCodexRealtimeVoice from /react and RealtimeVoice from /audio add a two-way WebRTC call on an existing Codex ACP session. The host supplies the published capability client's realtime methods, a session ID, admission state, and visible error reporting. codexRealtimeSignaling maps the SDP exchange through ACP; credentials and the backing agent remain owned by Codex. No new conversation is created and no second transcript store is written.

The call moves through idle, connecting, connected, and closing. Connected means negotiated media, not merely an accepted start request. A 30-second setup deadline reports failure. Ending a call immediately releases local media and then waits for remote stop confirmation. Changing the bound session, disabling the control, or unmounting cancels the call. Rejected startup, malformed signaling, transport loss, device removal, and playback refusal are visible failures; there is no retry or provider fallback. The host must not start dictation concurrently and must stop a call when its surface relinquishes ownership. A Codex account must support realtime; having the CLI installed alone is not sufficient.