@dtelecom/agents-js

AI voice agent framework for dTelecom rooms. Build real-time voice agents that join WebRTC rooms and interact with participants using speech-to-text, LLMs, and text-to-speech.

Architecture

Participant mic -> SFU -> server-sdk-node (Opus decode) -> PCM16 16kHz
  -> STT plugin -> transcription
  -> LLM plugin -> streaming text response
  -> Sentence splitter -> TTS plugin -> PCM16 48kHz
  -> AudioSource (48kHz Opus encode) -> SFU -> Participants

The pipeline uses a producer/consumer pattern: LLM tokens are split into sentences and queued, while a consumer synthesizes and plays audio concurrently. This minimizes time-to-first-audio.

After completed speech, the pipeline retains an 800 ms estimate for audio to reach the listener before changing its UI state to idle. This wait is cancellable: a new user turn releases it immediately, and an interrupted response or greeting does not add the wait after provider cleanup. Responses with no speech audio skip it entirely. Cancellation still flushes local buffered audio and waits for the active producer/consumer to finish before starting the queued turn; TTS synthesis remains sequential for providers that require it.

Cancellation covers warmup, conversation summarization, memory lookup and response generation. Cancelled summaries do not replace conversation history. A final transcript arriving during a response occupies a single pending-turn slot (the latest final replaces any earlier pending final) and interrupts the current cycle. The next cycle starts after the active provider tasks finish cancellation cleanup. An APIUserAbortError from an aborted cycle is expected cancellation; genuine provider failures remain errors.

Latency logs separate Queue (final transcript received to response-cycle start), Prepare (warmup, summary and memory lookup), LLM (request to first sentence, including TTFT) and TTS (first sentence to first audio). STT is server-reported processing time. Overall includes those phases but excludes STT endpointing wait; it is unavailable for turns without a text response. These measurements are not microphone-to-speaker latency.

For dTelecom TTS, cancellation sends clear and waits for cleared before reusing the WebSocket. Late audio from the cancelled request is discarded. A missing acknowledgement closes the connection after five seconds so a subsequent request reconnects; this is an error timeout, not a delay applied to successful cancellations.

Install

npm install @dtelecom/agents-js @dtelecom/server-sdk-js @dtelecom/server-sdk-node

Quick Start (other providers)

import { VoiceAgent, setLogLevel } from '@dtelecom/agents-js';
import { DeepgramSTT, OpenRouterLLM, CartesiaTTS } from '@dtelecom/agents-js/providers';

const agent = new VoiceAgent({
  stt: new DeepgramSTT({ apiKey: process.env.DEEPGRAM_API_KEY! }),
  llm: new OpenRouterLLM({
    apiKey: process.env.OPENROUTER_API_KEY!,
    model: 'openai/gpt-4o',
  }),
  tts: new CartesiaTTS({
    apiKey: process.env.CARTESIA_API_KEY!,
    voiceId: 'your-voice-id',
  }),
  instructions: 'You are a helpful voice assistant.',
});

await agent.start({
  room: 'my-room',
  apiKey: process.env.DTELECOM_API_KEY!,
  apiSecret: process.env.DTELECOM_API_SECRET!,
});

apiKey and apiSecret are a dTelecom Cloud project's API key; the agent's token comes from the cloud at DTELECOM_CLOUD_URL. Pass cloud instead to reuse a client (see below).

dTelecom Cloud speech (0.7.0+)

With a dTelecom Cloud project, give the agent and the dTelecom providers one CloudClient from @dtelecom/server-sdk-js 4.1 or newer. The agent mints its own WebRTC token; the providers open their speech sessions, link them to the agent's room session, and extend them before they run out.

import { CloudClient } from '@dtelecom/server-sdk-js';
import { VoiceAgent } from '@dtelecom/agents-js';
import { DtelecomSTT, DtelecomTTS } from '@dtelecom/agents-js/providers';

// DTELECOM_API_KEY, DTELECOM_API_SECRET and DTELECOM_CLOUD_URL from the environment
const cloud = new CloudClient();

const agent = new VoiceAgent({
  stt: new DtelecomSTT({ cloud, language: 'en', partials: true, smartTurn: true }),
  tts: new DtelecomTTS({ cloud, voices: { en: { voice: 'af_heart', langCode: 'a' } } }),
  llm, instructions: 'Be helpful and concise.',
});
await agent.start({ room: 'voice-demo', identity: 'assistant', cloud, ttl: '2h' });

Your backend gives each human participant a token with cloud.createToken. How the sessions work:

• STT: one session per participant stream, billed per connected minute. Each holds 10 minutes of balance and is extended by 10 minutes when under 2 remain. Change it with cloudSession: { minutes, extendWhenMinutesLeft }.
• TTS: one session for the agent's voice, billed per character. It holds 2,000 characters and is extended before a sentence would leave fewer than 500, and before its 30-minute window closes. Change it with cloudSession: { characters, extendWhenCharactersLeft }.
• Unused holds are released when a session ends. An extension sends the new key to the open socket; a reconnect uses the latest key. A session the cloud refuses (for example insufficient_balance) surfaces as an error event.
• ttl sets the agent's own room session (default 6 hours, at most 24).

Rates are in the dashboard's billing catalog: STT $0.004 per connected minute, TTS $0.006 per 1,000 characters, WebRTC audio $0.0001 per participant minute. File transcription, WAV synthesis, translation, vision and moderation are on CloudClient itself; VoiceAgent does not call them.

dTelecom gateway speech (x402)

Create a prepaid bundle with DtelecomGateway from @dtelecom/x402-client, then pass its tokens to the providers. This example assumes gateway is configured with a funded wallet and llm implements the LLM plugin interface:

import { VoiceAgent } from '@dtelecom/agents-js';
import { DtelecomSTT, DtelecomTTS } from '@dtelecom/agents-js/providers';

const session = await gateway.createAgentSession({
  roomName: 'voice-demo', participantIdentity: 'assistant',
  clientIdentity: 'listener', durationMinutes: 5, language: 'en',
});
const agent = new VoiceAgent({
  stt: new DtelecomSTT({
    serverUrl: session.stt.serverUrl, sessionKey: session.stt.token,
    language: 'en', partials: true, smartTurn: true,
  }),
  tts: new DtelecomTTS({
    serverUrl: session.tts.serverUrl, sessionKey: session.tts.token,
    voices: { en: { voice: 'M5', langCode: 'en' }, ru: { voice: 'M5', langCode: 'ru' } },
    speed: 1.1, // application preference; SDK default is 1.0
    // watermark: true, // independent explicit opt-in
  }),
  llm, instructions: 'Be helpful and concise.',
});
await agent.start({ room: 'voice-demo', token: session.webrtc.agent.token, wsUrl: session.webrtc.agent.wsUrl });

The client joins with session.webrtc.client. STT is $0.004/connected session minute; TTS is $0.006/1,000 spoken Unicode code points; WebRTC is $0.0001/audio participant minute. Two participants, one STT session and 367 TTS characters/minute cost approximately $0.0064/minute, excluding LLM charges. Live pricing and enabled services.

This framework runs live voice agents. Translation, file transcription/diarization, WAV jobs, vision and moderation use the separate x402 client SDK; they are not automatically invoked by VoiceAgent.

Plugin Interfaces

STT

interface STTPlugin {
  createStream(options?: STTStreamOptions): STTStream;
}

interface STTStream {
  sendAudio(pcm16: Buffer): void;
  on(event: 'transcription', cb: (result: TranscriptionResult) => void): this;
  on(event: 'error', cb: (error: Error) => void): this;
  close(): Promise<void>;
}

LLM

interface LLMPlugin {
  chat(messages: Message[], signal?: AbortSignal): AsyncGenerator<LLMChunk>;
  warmup?(systemPrompt: string): Promise<void>;
}

TTS

interface TTSPlugin {
  synthesize(text: string, signal?: AbortSignal): AsyncGenerator<Buffer>;
  warmup?(): Promise<void>;
}

dTelecom partial transcripts and Smart Turn

import { DtelecomSTT } from '@dtelecom/agents-js/providers';

const stt = new DtelecomSTT({
  serverUrl: session.stt.serverUrl,
  sessionKey: session.stt.token,
  language: 'en',
  partials: true,
  smartTurn: true, // independent opt-in; omit for ordinary acoustic endpointing
});

Both options are omitted by default, preserving existing final-only behavior. partials requests replaceable recognition windows. The provider sends smartTurn as smart_turn; it requires a configured server model and supported ANE VAD language route. These choices survive language changes and reconnects.

With this provider in VoiceAgent, subscribe to:

• transcription_partial: { text, isFinal: false, utteranceId, revision, windowStartSeconds, provisional: true, replacePrevious: true, speaker }. Replace the displayed provisional window; do not append it to committed text.
• partial_end: { utteranceId, reason, speaker }. Clear provisional text on finalizing, discarded, replaced or reconfigured.
• transcription: partials also appear here with isFinal: false; finals carry the same utteranceId and supersede them. Only finals start an LLM turn. Nonempty partials can trigger barge-in while the agent is speaking.
• stt_ready: { clientId?, language?, smartTurn?, speaker }. smartTurn: "active" confirms the semantic hook; "acoustic_fallback" reports silence detection. An absent field does not confirm activation. Missing server-model errors surface through error.

Raw STTStream exposes the same events without speaker, with ready instead of stt_ready. Providers other than dTelecom may not emit these optional event types. For transcript display, subscribe to either transcription_partial or nonfinal transcription, not both.

Smart Turn defaults to off. Current server defaults are completion threshold 0.5, minimum pause 500 ms, maximum 2 seconds of received silence and classifier deadline 250 ms. These are not end-to-end latency guarantees. Use server-sdk-node 0.2.6 or later to supply silence during Opus DTX/input stalls. See Smart Turn integration notes for current behavior and separately dated historical acceptance results.

Configuration

Events

Custom Providers

Implement the plugin interface and pass it to VoiceAgent:

import { BaseSTTStream, type STTPlugin, type STTStreamOptions } from '@dtelecom/agents-js';

class MySTTStream extends BaseSTTStream {
  sendAudio(pcm16: Buffer): void {
    // Send audio to your STT service
  }

  async close(): Promise<void> {
    // Clean up
  }
}

class MySTT implements STTPlugin {
  createStream(options?: STTStreamOptions) {
    return new MySTTStream();
  }
}

const agent = new VoiceAgent({
  stt: new MySTT(),
  // ...
});

Data Channels

Receive data channel messages from participants:

const agent = new VoiceAgent({
  // ...
  onDataMessage: (payload, participantIdentity, topic) => {
    const message = JSON.parse(new TextDecoder().decode(payload));
    console.log(`${participantIdentity} sent:`, message);
  },
});

Complete-request SSML (0.5.0+)

DtelecomTTS.synthesizeSSML(document, signal) negotiates dtelecom-ssml-v1 with the server and sends one complete document unchanged. It does not use the legacy language-tag splitter or insert language-switch silence. Servers must explicitly enable the profile; unsupported servers are rejected before markup is sent for synthesis.

For greetings or announcements, use:

await agent.saySSML(
  '<speak>Hello.<break time="400ms"/>Welcome back.</speak>',
  'Hello. Welcome back.',
);

The second argument supplies plain display and conversation-context text. Barge-in uses the same cancellation path as say(). Ordinary say() and streamed LLM responses remain plain text. Incremental SSML and word-timing marks are not implemented. Server-side validation and expanded-text metering are authoritative; see the public SSML contract. One document stays within one engine, has a 60-second output bound, and uses effective speed 0.75–1.5. No additional SSML price multiplier applies.

OpenRouter model controls

For providers that require explicit reasoning settings or reject sampling parameters:

new OpenRouterLLM({
  apiKey: process.env.OPENROUTER_API_KEY!,
  model: 'openai/gpt-6-luna',
  reasoningEffort: 'none',
  temperature: null, // omit the temperature parameter
  providerRouting: { sort: 'latency', requireParameters: true },
});

These controls also apply to plain-text summarization requests. Existing callers retain the 0.7 temperature default and omit reasoning effort unless configured.

Optional TTS watermark (0.6.0+)

AudioSeal B is enabled only when the SDK user explicitly requests it:

const tts = new DtelecomTTS({
  serverUrl,
  sessionKey,
  voices: { en: { voice: 'M5', langCode: 'en' } },
  watermark: true,
});

Omitting watermark or setting it to false keeps existing unmarked speech. The option applies to plain text and synthesizeSSML() independently of STT Smart Turn/partials. The server must enable the optional feature; the provider checks availability during warmup/first use, caches it per connection, and requires watermark: "active" before yielding PCM. It logs TTS watermark=active (AudioSeal B). Unsupported servers or failed activation produce an error, never silent unmarked output.

The accepted B configuration runs on 48 kHz float audio after TTS resampling and SSML gain, before PCM16/Opus. It adds an 80 ms input buffer and approximately 5 ms inference in the M6 lab; live timing depends on load and audio arrival. No filter or resampling is applied after watermarking. Short/interrupted speech may not remain detectable. This option is off by default even on capable servers.

License

Apache-2.0