@animated-waffle/client

Framework-independent browser SDK for embedding a published Narya talking Agent: connection lifecycle, microphone input, text input, state, and transcripts. React applications should use @animated-waffle/react, which re-exports this entire API and adds the Avatar renderer. For plain JavaScript or another browser framework, use @animated-waffle/web and mountWaffleAvatar. This package is headless; it does not render a character.

Install the current npm release without copying a mutable registry version:

pnpm add @animated-waffle/client@latest

The managed-session identity and sessionContext APIs described here were introduced in version 0.3.1. The workspace package.json is the source of truth for this repository's version; @latest resolves the current npm release.

The Agent owns its avatar, voice, and prompt. Media runtime, API routes, and avatar asset URLs are SDK implementation details. Production is the default API origin; tests and non-production integrations may pass apiOrigin.

Connect

Keep the permanent workspace API key on your server. Exchange it for a short-lived Agent session token and give the browser a callback that obtains one for each connection:

import { WaffleClient } from '@animated-waffle/client'

const agentId = '22222222-2222-4222-8222-222222222222'
const endUserId = '55555555-5555-4555-8555-555555555555'
const waffle = new WaffleClient({
  agentId,
  // Set false for text-triggered, output-only applications.
  inputAudioEnabled: false,
  sessionContext: {
    userName: ' Space é ',
    timeZone: 'Asia/Shanghai',
  },
  sessionToken: async () => {
    const response = await fetch('/api/narya-session-token', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ agentId, endUserId }),
    })
    const body = await response.json()
    return body.token
  },
})

waffle.on('transcript', ({ id, role, text, final }) => {
  // Completed Narya conversation rows use the same id as persisted history.
  if (final) console.log(id, role, text)
})
waffle.on('transcriptProgress', ({ segmentId, role, text, complete }) => {
  // Ephemeral ASR/playout caption; never persist segmentId as a message id.
})

await waffle.connect()

endUserId is one stable bare UUID per end user. An anonymous application can generate it once with crypto.randomUUID() and persist it on that device.

Region

Pass region: 'jp' to new WaffleClient({ agentId, sessionToken, region: 'jp' }) or client.connect({ region: 'jp' }). The SDK selects both the Japan API and Agent; 'us' selects the US. A connection option overrides the constructor default. Disconnect before switching an active session. Omitted regions keep existing behavior.

For DEV, add apiOrigin: 'https://dev.aniwaffle.com'; region selection stays in DEV. Custom API origins are preserved. Custom startSession callbacks receive region and must forward it to their backend. See the region guide.

Creator Developer Feedback

Developer Feedback is opt-in and unavailable unless the application supplies a callback to its own creator-authenticated backend. The SDK invokes it lazily when feedback is opened or saved, after the API has returned the persisted source Session id:

const waffle = new WaffleClient({
  agentId,
  sessionToken: () => getSessionToken(agentId),
  developerFeedback: ({ agentId, sourceSessionId, signal }) =>
    fetch('/api/narya-developer-feedback-grant', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ agentId, sourceSessionId }),
      signal,
    }),
})

That backend response uses the public grant shape { endpoint, token, expires_at, source_session_id }. It must be protected as a creator development route and mint the grant with a server-side workspace key; never expose that key to the browser. The feedback endpoint receives only the five-minute grant with credentials: 'omit'. It cannot use the ordinary Agent session token. Non-loopback SDK development origins must be included in the API's configured origin allowlist.

loadDeveloperFeedback() reads the current private draft. submitDeveloperFeedback() appends one Preserve/Change observation for the whole Session or a persisted Assistant message. It never submits the round, generates a revision, or publishes the Agent. Reuse an explicit feedback id when retrying after an ambiguous network failure.

To target a non-production environment:

const waffle = new WaffleClient({
  agentId,
  sessionToken: () => getSessionToken(agentId),
  apiOrigin: 'https://animated-waffle-dev.narya.ai',
})

API surface

• sessionContext?: { userName?: string; timeZone?: string } — available only with the default sessionToken starter. Nonblank values are sent exactly as provided; blank values are omitted. It is mutually exclusive with the advanced startSession callback, so token-path consumers do not recreate the raw managed-session request.
• reportedSource?: 'Web' | 'SDK' | 'Client' | 'Unknown' — self-reported integration family for observability only. The SDK defaults to SDK.
• connect({ inputMode?, sessionId?, overrides? }) — 'push-to-talk' (default) or 'automatic'; resolves only when the Agent runtime is ready.
• session.agentStudioUrl? — server-authored dashboard target for the exact Agent and source Session. Development UI may show “Open Agent in Studio” when present; do not rebuild it from avatarId. Older protocol-v2 servers may omit it during rolling upgrades.
• inputAudioEnabled: false — never requests or enables the browser microphone; text input, Agent audio, transcripts, and reconnects continue to work. Push- to-talk controls fail closed with input_audio_disabled. The default is true, enabled only after the Agent is ready.
• muted — silences the Agent's voice while it keeps performing: the session, the transcript and the Avatar's cues carry on, only playback goes quiet.
• audioJitterBufferMs: 20 — preferred native WebRTC audio buffer, in milliseconds (default 20; use null for browser defaults). The browser still adapts to network jitter. Unsupported browsers keep their defaults.
• audioPlayback: false — the client builds no audio element of its own. Play remoteAudioTrack yourself; an application that routes the voice through its own element, mixer or Web Audio graph must turn this off, or the voice is heard twice. muted then has nothing to do.
• disconnect(), sendText(text, { audioResponse? }). audioResponse defaults to true; false keeps the typed turn and transcript but suppresses reply audio for that turn.
• loadDeveloperFeedback(), submitDeveloperFeedback(input) — available only with the explicit creator BFF callback above.
• startPushToTalk() / commitPushToTalk() / cancelPushToTalk().
• on('connectionState' | 'session' | 'remoteAudioTrack' | 'transcriptProgress' | 'transcript' | 'agentSpeaking' | 'declined' | 'error', fn) — declined carries the operator-facing reason for deliberate silence. The call returns the unsubscribe function.
• Conversation transcripts are completed persisted rows, carry the same required id as history, and currently always have final: true. Provider sentence/progress framing never defines a public conversation row.
• Transcript progress is ephemeral and keyed by a runtime segmentId; user ASR and assistant playout update it incrementally until complete closes it.
• connectionState, session, remoteAudioTrack getters; status is the deprecated 0.1 alias. Failures throw WaffleError with code and a typed stage: session, transport, audio, transcript, avatar-load, or avatar-renderer. A failed connect() emits and rejects with the same error instance.

./internal is a private contract consumed by @animated-waffle/react; it is not part of the public API and may change without notice.

See the SDK documentation for the complete integration guide.

Application sessions

For an application-owned conversation, use application: true with a startSession callback returning the media session created by WaffleServer.startApplicationSession. The backend retains the separate control grant; the browser never executes tools or forwards privileged callbacks. See the application-session contract.

Development diagnostics

client.diagnostics.server.waitDetails splits input handoff to LLM start (negative when preemptive), first token to TTS request (sentence buffering plus connection), TTS connection, first PCM to worker audio sink, and the original PCM's leading-silence estimate. Automatic ASR finalization and end-of-turn decision are separate server.asrFinalizationMs and server.endOfTurnDecisionMs fields. Older workers omit these fields.

For RTC playback, client.diagnostics.turn.transport contains the selected browser-to-LiveKit ICE RTT, jitter-buffer/target and device-playout averages over the first 250 ms after detected playback, and sampleWindowMs. Sampling completes asynchronously and preserves the original playback timestamp; audio never waits for diagnostics. Unsupported, reset or stale counters stay null. These are overlapping diagnostic samples, not additive one-way network latency; server and client observations may describe different inputs. They contain no audio, transcripts, device identifiers or network addresses. The built-in Web/React diagnostics panel renders the same fields.

Read client.diagnostics and subscribe with client.on('diagnostics', render). React returns the same snapshot as waffle.diagnostics. The React useWaffle and web mountWaffleAvatar adapters show a collapsed floating diagnostics panel in development builds; production defaults off. Their diagnostics option overrides visibility. For an inline view, use WaffleDiagnosticsPanel or mountWaffleDiagnostics(host, client) and disable the automatic overlay. The headless client never adds UI. Mounting a panel never starts a connection or microphone.

The panel shows connection/media/Avatar state, connection phases, latest client input-end-to-playback estimate, safe errors and the current Session Studio link. Updated workers additionally report ASR, LLM, TTS and worker-output timings using Studio's reducer. Client and server rows have separate correlation IDs and clocks; do not sum the stages or treat a worker output as browser playback. Missing data remains unavailable. Voice input activity and RTC/paired output timing are estimates, not calibrated acoustic measurements. Snapshots exclude text, audio, tokens and raw error messages. The panel is optional; existing timing telemetry is independent of it.

Microphone capture

audioCapture accepts native LiveKit capture constraints (excluding custom processors). They apply before the first microphone opens, and after reconnecting. They stay local to the client; authentication, billing and server voice routing are unchanged.

const client = new WaffleClient({
  agentId,
  sessionToken: () => fetchSessionToken(),
  // Only for an isolated virtual input or audio that already has echo processing.
  audioCapture: { echoCancellation: false },
})

Keep echo cancellation enabled for normal speaker/microphone setups. Disabling it can feed the Agent's voice back into recognition and trigger unwanted interruptions. No device-name detection or automatic processing changes are applied. Inspect client.microphoneTrack?.getSettings() after connecting to check what the browser actually enabled; changing an already-open track may be rejected. useWaffle accepts the same option. Equivalent inline constraints retain the connection; changing their values recreates the client. Capture options do not change LiveKit's server-side interruption warmup or end-of-turn settings.

Microphone controls and session event logs

client.microphone contains the current input label, local device ID, available inputs, mute/enabled flags and switch status. Labels may be empty until the browser grants capture permission. refreshInputDevices() only enumerates; it never requests permission. setInputDevice(deviceId) switches a connected call, cancels an active PTT hold and rebinds capture and level observation. A failed switch rejects without ending the call. The selected device is reused when this client reconnects. Device lists update on browser devicechange.

client.on('microphone', renderInputSelector)
client.on('inputLevel', renderMeter) // { rms, peak } in [0, 1], at most 20 Hz; null = unavailable
client.on('inputIssue', renderInputNotice) // null clears the current notice
await client.setInputDevice(selectedDeviceId)

The meter measures energy, not recognized speech. It also runs for applications that own playback (audioPlayback: false). It does not open a second microphone.

client.on('diagnosticEvent', handler) receives structured session events; client.diagnosticHistory retains the latest 128 for local debugging, including failed connection attempts. Compatible workers automatically store these events under the authenticated session. No operator-only instrumentation is required. Events include connection/reconnection, explicit disconnect, microphone/track state and actual device label, device changes, PTT press/release/cancel, control acknowledgements/failures, PCM completion/bytes, 5-second energy summaries, transcript lengths, remote audio, playback mute, interruption and input issues. Device IDs stay local; audio, text content, credentials and URLs are not included in client reports. Server workflow logs separately retain bounded model output.

Delivery is best effort: up to 64 unsent events, 8 per batch, stable IDs for retry deduplication and at most 4,096 events per connection. Three consecutive send failures stop retries for that connection. Lost connectivity or a session that never reaches a compatible worker can leave events only in local history; subscribe to diagnosticEvent to send these through your application's logging. Client elapsed times use a separate clock from server event timestamps.