@ depgram/ 代理服务器

核心 SDK 用于 Deepgram 语音代理 API。管理WebSocket会话,麦克风抓取,以及音频回放的音量/频率分析.

状态

此包为前1.0. 界面在小版本之间可能会有变化.

安装

npm install @deepgram/agents

或与 Bun 连接 :

bun add @deepgram/agents

快速启动

import { AgentSession, AgentMicrophone, AgentPlayer } from "@deepgram/agents";

const session = new AgentSession({
  auth: { tokenFactory: () => fetch('/api/deepgram-token').then(r => r.text()) },
  agent: {
    listen: { provider: { type: 'deepgram', version: 'v1', model: 'nova-3' } },
    think: { provider: { type: 'open_ai', model: 'gpt-4o-mini' } },
    speak: { provider: { type: 'deepgram', model: 'aura-2-thalia-en' } },
  },
  audio: { output: { encoding: 'linear16', sampleRate: 24_000 } },
});

const player = new AgentPlayer();
session.on("audio", (chunk) => player.queue(chunk));
session.on("conversation-text", (msg) => {
  console.log(`${msg.role}: ${msg.content}`);
});

const mic = new AgentMicrophone((data) => session.sendAudio(data));

await session.connect();
await mic.start();

// Later:
// mic.stop();
// session.disconnect();
// player.dispose();

认证

通过两个认证模式 AgentSessionConfig.auth数字 :

// Server-side: raw API key
{ apiKey: "your-deepgram-api-key" }

// Browser-safe: token factory (recommended)
{ tokenFactory: () => fetch('/api/token').then(r => r.text()) }

在每次连接和再连接尝试之前,都叫"代币工厂". 托肯被缓存到被再连接所失效,因此工厂并不是在每个音频帧上调用.

SDK使用 Sec-WebSocket-Protocol 页眉诡计 -- -- 浏览器 WebSocket 构造器不支持自定义信头,因此该符被传递为子标值. 这是内部处理的.

代理会议

核心WebSocket会话. 环绕 @deepgram/sdk与 :

• Token 工厂认证(每次重新连接的新证书)
• 类型事件
• 与 jitter 重接的指向后退
• 自动保存Alive pings
• 音频缓冲到 SettingsApplied
• SDK 二进制命令正常化 Blob 信件给 ArrayBuffer

@deepgram/agents 直接取决于 @deepgram/sdk。它会禁用基本的套接字重试,并拥有应用程序级重接,这样每次尝试都可以刷新认证并恢复设置。 仅为内置代理配置而恢复累积的对话上下文;协议不能将内置上下文附加到代理 UUID上. 消费者继续收到文件 ArrayBuffer 从 audio 事件,即使基本的 SDK 发送二进制消息给它的回调为 Blob。 。 。 .

构造器

const session = new AgentSession(config: AgentSessionConfig);

配置

interface AgentSessionConfig {
  auth: { apiKey: string } | { tokenFactory: () => Promise<string> };
  agent: AgentSettingsObject | string;  // inline config or pre-built agent UUID
  audio?: {
    input?: { encoding?: AudioEncoding; sampleRate?: number };   // default: linear16 @ 16kHz
    output?: { encoding?: OutputEncoding; sampleRate?: number }; // default: linear16 @ 24kHz
  };
  keepAliveInterval?: number;  // default: 10_000ms
  reconnect?: ReconnectConfig;
  experimental?: boolean;
  tags?: string[];
}

方法

活动

session.on("welcome", (msg) => {});
session.on("settings-applied", (msg) => {});
session.on("conversation-text", (msg) => {});
session.on("user-started-speaking", (msg) => {});
session.on("agent-thinking", (msg) => {});
session.on("agent-started-speaking", (msg) => {});
session.on("agent-audio-done", (msg) => {});
session.on("function-call-request", (msg) => {});
session.on("function-call-response", (msg) => {});
session.on("prompt-updated", (msg) => {});
session.on("speak-updated", (msg) => {});
session.on("think-updated", (msg) => {});
session.on("listen-updated", (msg) => {});
session.on("latency-report", (msg) => {});
session.on("history", (msg) => {});
session.on("injection-refused", (msg) => {});
session.on("error", (msg) => {});
session.on("warning", (msg) => {});

// Binary audio from the agent
session.on("audio", (chunk: ArrayBuffer) => {});

// SDK lifecycle
session.on("connecting", () => {});
session.on("connected", () => {});
session.on("reconnecting", (attempt, delayMs) => {});
session.on("disconnected", (reason) => {});
session.on("sdk-error", (err) => {});

状态

session.state; // "idle" | "connecting" | "connected" | "reconnecting" | "disconnected"

重新连接

自动重联是默认启用的,带有指数回放+jitter. 之后 SettingsApplied,每个运行时间设置的最新值(即时,听取,说话,思考)会被重放,按每次设置最后一次更新时排序,再连接时的音频队列被冲出. 仅为内置代理配置恢复已累积的文本和已完成的函数调用上下文. 在意外的关闭或出错后发送的音频会停留在尝试中队列并仅在替换连接收到后才恢复 SettingsApplied。连续尝试计数器也在 SettingsApplied不仅仅是在运输开通时 通过 reconnect数字 :

{
  enabled: true,       // default
  maxAttempts: 8,      // default
  baseDelay: 500,      // ms, default
  maxDelay: 30_000,    // ms, default
  jitter: true,        // default: +/-20%
}

麦克风代理

通过AudioWorklet从用户的麦克风中捕获PCM音频.

使用量

const mic = new AgentMicrophone(
  (data: ArrayBuffer) => session.sendAudio(data),
  {
    sampleRate: 16_000,         // default
    echoCancellation: true,     // default
    noiseSuppression: true,     // default
    autoGainControl: true,      // default
  },
);

await mic.start();
mic.mute();
mic.unmute();
mic.stop();

音量和频率数据

mic.getInputVolume();            // 0-1, RMS-based
mic.getInputByteFrequencyData(); // Uint8Array of frequency bin magnitudes (0-255)

活动

mic.on("audio-frame", (data: ArrayBuffer) => {});
mic.on("error", (err: Error) => {});

代理服务器

解码并播放原始 PCM Int16 ()linear16) 代理的音频. 提供可视化的体积/频率分析,并支持通过 interrupt()。 。 。 . AgentPlayer 不解码压缩输出; 使用 linear16 输出或提供单独的解码器/播放器.

使用量

const player = new AgentPlayer({ sampleRate: 24_000 }); // default

// Queue audio from the session
session.on("audio", (chunk) => player.queue(chunk));

// Barge-in: interrupt when the user starts speaking
session.on("user-started-speaking", () => player.interrupt());

// Volume control
player.setVolume(0.8);
player.mute();
player.unmute();

// Cleanup
player.dispose();

音量和频率数据

player.getOutputVolume();            // 0-1, RMS-based
player.getOutputByteFrequencyData(); // Uint8Array of frequency bin magnitudes (0-255)
player.getRemainingPlaybackTime();   // seconds of queued audio remaining

出口

// Classes
export { AgentSession, AgentMicrophone, AgentPlayer };

// Types
export type {
  AgentState,
  AgentSessionConfig, AuthConfig, TokenFactory, ReconnectConfig,
  AgentSessionEvents,
  MicrophoneOptions,
  PlayerOptions,
  AgentSettingsObject, ListenSettings, ThinkSettings, SpeakSettings,
  AgentMessageBehavior,
  // Server messages
  WelcomeMessage, SettingsAppliedMessage, ConversationTextMessage,
  UserStartedSpeakingMessage, AgentThinkingMessage,
  FunctionCallRequestMessage, FunctionCallResponseMessage, FunctionCallItem,
  AgentStartedSpeakingMessage, AgentAudioDoneMessage,
  PromptUpdatedMessage, SpeakUpdatedMessage, ThinkUpdatedMessage,
  ListenUpdatedMessage, LatencyReportMessage, HistoryMessage,
  AgentErrorMessage, AgentWarningMessage,
  InjectionRefusedMessage, ServerMessage,
};

许可证

麻省理工学院