@alexkroman1/aai

The AssemblyAI Agent SDK: everything an agent.ts file imports. The self-hostable runtime the CLI and the managed platform run is @alexkroman1/aai-runtime.

npm i @alexkroman1/aai zod

Most projects don't install this directly — aai init (from @alexkroman1/aai-cli) scaffolds a project with it wired up.

The simple case stays simple

One mode of the SDK is a prompt, a voice and some keyterms over a single socket — the whole agent:

import { agent, assemblyAIS2s } from "@alexkroman1/aai";

export default agent({
  name: "Support",
  // The prompt is the system-prompt.md beside this file.
  mode: "s2s",
  s2s: assemblyAIS2s({ voice: "jane", keyterms: ["AssemblyAI"] }),
});

Everything below is what becomes available when your own code runs alongside it. Nothing here is required to get a working agent.

Defining an agent

agent.ts — the definition, plus the slot that owns this session's state:

import { agent, sessionSlot } from "@alexkroman1/aai";

export const notesSlot = sessionSlot("notes", () => ({ items: [] as string[] }), {
  view: (notes) => ({ count: notes.items.length }),
  sync: true,
});

// `sync: true` on the slot is what shows it to the browser client (read there
// with `useAgentState`); the prompt is the system-prompt.md beside this file.
export default agent({ name: "Notes" });

• agent() — the agent definition; every field and default is documented on AgentDef. With no provider fields it runs the default all-AssemblyAI STT → LLM → TTS pipeline, billed to one ASSEMBLYAI_API_KEY; voice: "aai-michael" picks its TTS voice.
• tool() — a typed tool for the stateless case: Zod inputSchema and an execute(args, ctx) that runs server-side with ctx.env (secrets), ctx.signal (aborts on barge-in), ctx.generate (one-shot LLM calls), ctx.delegate (subagents), ctx.workflows (start and find durable runs) and ctx.send (push events to the browser client).
• sessionSlot() — a typed named slot owning a session's state. slot.tool() reads it (the value is deeply frozen) and slot.updateTool() writes it synchronously. There is no ctx.state and no ctx.db.
• dialog() / procedure() — a flow the model is gated by, and a flow your own code drives.
• speaker() / roster() — a second voice: delegated to off the line (ctx.delegate), or handed the call (speaks: true).
• workflow() / workflowApp() — a durable run, and a form-shaped agent whose front door is a workflow rather than a microphone.
• assemblyAIPipeline() — the same default pipeline as an explicit spread (...assemblyAIPipeline({ region: "eu" })), for when you want the three stages visible in the config or an EU region across STT and the gateway.

agent() takes one object. AgentDef is the reference for what each field MEANS — every field and default is documented there — and AgentParams is the reference for which combinations are LEGAL. A fuller configuration, with the fields the example above leaves out:

import { agent, sessionSlot } from "@alexkroman1/aai";
import { assemblyAITts } from "@alexkroman1/aai/tts";

// `sync: true` is what the browser client renders with `useAgentState`.
const cart = sessionSlot("cart", () => ({ items: [] as string[] }), {
  view: (c) => ({ count: c.items.length }),
  sync: true,
});

export default agent({
  name: "Storefront",
  greeting: "Storefront here — what are you after?",
  tts: assemblyAITts({ voice: "aai-michael" }),
  // Server-side helpers the model may call, on top of your own tool files.
  builtinTools: ["calculate"],
  // Tool-calling steps per reply, and how long a pause ends the caller's turn.
  maxSteps: 6,
  turnTaking: { minSilenceMs: 1200 },
  // Observe-only hooks over the session event stream.
  events: {
    "tool.called": (event) => {
      console.log("called", event.toolName);
    },
  },
});

Tools: a tool is a FILE

An agent's value is the actions it takes, and every one of them lives in a system only your code can reach. A tool is a file in tools/, named by its own filename — agent() takes no tools field and nothing registers it:

// file: agent.ts
import { agent, sessionSlot } from "@alexkroman1/aai";

export const notesSlot = sessionSlot("notes", () => ({ items: [] as string[] }));

export default agent({ name: "Notes" });

// file: tools/add_note.ts
import { z } from "zod";
import { notesSlot } from "../agent.ts";

export default notesSlot.updateTool({
  description: "Save a note for the caller",
  inputSchema: z.object({ text: z.string() }),
  // `updateTool` hands the body a mutable draft, stored when it returns.
  execute: ({ text }, notes) => {
    notes.items.push(text);
    return { saved: notes.items.length };
  },
});

Secrets stay out of the code, and in-flight work stops the moment the caller interrupts — pass ctx.signal to anything slow:

import { requireEnv, tool } from "@alexkroman1/aai";
import { z } from "zod";

export default tool({
  description: "Get current weather for a city",
  inputSchema: z.object({ city: z.string() }),
  execute: async ({ city }, ctx) => {
    const key = requireEnv(ctx, "WEATHER_KEY");
    const resp = await fetch(`https://api.example.com/weather?q=${city}&key=${key}`, {
      // Aborts on barge-in, reset, session stop, or this call's timeout.
      signal: ctx.signal,
    });
    return await resp.json();
  },
});

Built-in tools and code execution

builtinTools opts the model into server-side helpers it can call — web_search, visit_webpage, get_page_design, fetch_json, think, remember/recall, calculate, and run_code, where the model writes JavaScript and runs it in the agent's sandbox. Models are unreliable at mental arithmetic; real execution is what makes the number right on a call where the caller is about to act on it.

import { agent } from "@alexkroman1/aai";

// system-prompt.md: "Use run_code for ANY math, counting, or data processing."
export default agent({
  name: "Analyst",
  builtinTools: ["web_search", "run_code"],
});

Omitting builtinTools enables none of them. The network-facing builtins take model-controlled URLs, so they are SSRF-screened outside a container and cap what comes back; run_code needs a sandbox — the platform's, or a zero-permission Deno under AAI_RUN_CODE=deno — and refuses outside one.

Typed session state

Long calls accumulate facts — the order number, what's been verified, what's in the cart. Without structured state the agent re-derives them from the transcript every turn. sessionSlot() is one declaration in a shared module, and the only way to keep state across a session's tool calls:

import { sessionSlot } from "@alexkroman1/aai";
import { z } from "zod";

export const cartSlot = sessionSlot("cart", () => ({ items: [] as string[] }));

// `slot.tool()` READS — what the body is handed is frozen.
export const listCart = cartSlot.tool({
  description: "List what's in the cart",
  execute: (_args, cart) => ({ items: cart.items }),
});

// `slot.updateTool()` WRITES — mutate the draft; it is stored when the body
// returns, so a write is SYNCHRONOUS and may not `await`.
export const addItem = cartSlot.updateTool({
  description: "Add an item to the cart",
  inputSchema: z.object({ item: z.string() }),
  execute: ({ item }, cart) => {
    cart.items.push(item);
    return { count: cart.items.length };
  },
});

Slots hold plain data (a Map, Set, Date or class instance is refused with the field named). On the platform they are stored for you — a crash or a redeploy no longer loses them, and there is nothing to enable. Under aai dev they live in memory unless you set a DATABASE_URL; the code is the same either way.

Every session can resume

Connections drop, and five minutes of identity verification plus a half-built order shouldn't restart. A client reconnecting with ?sessionId=<id> rejoins the same session: the event log is restored, slot values are rehydrated, and the greeting is suppressed because the caller has already heard it. A resume that finds nothing is treated as a new session and greets, rather than sitting connected and silent. Background work is found again by key — on a later turn, or on the next call:

import { tool, workflow } from "@alexkroman1/aai";
import { z } from "zod";

export const research = workflow({
  description: "Research a topic",
  input: z.object({ topic: z.string() }),
  run: async (input) => ({ topic: input.topic, findings: [] as string[] }),
});

export default tool({
  description: "Check on research from earlier",
  execute: async (_args, ctx) => {
    // The run outlived the call — find it by the key it was started with.
    const [run] = await ctx.workflows.find(research, ctx.sessionId);
    if (!run) return { status: "none started" };
    return run.status === "completed" ? run.output : { status: run.status };
  },
});

Conversation order as a guarantee

"Verify before disclosing" in a system prompt is advice. dialog() makes it a rule: a tool declared with when simply does not run outside those states, and the model gets back a ToolFailure naming what has to happen first. The position is a slot underneath, so it survives a reconnect.

import { dialog } from "@alexkroman1/aai";
import { z } from "zod";

export const claim = dialog("claim", {
  initial: "verifying",
  states: {
    verifying: {
      instruction: "Verify the caller's policy number before quoting anything.",
      on: { VERIFIED: "quoting" },
    },
    quoting: {
      instruction: "Quote the excess. Do not re-verify.",
      on: { QUOTED: "done" },
    },
    done: { final: true },
  },
});

// Cannot run before the caller is verified; advances the dialog when it does.
export const quoteClaim = claim.tool({
  when: "quoting",
  send: { type: "QUOTED" },
  description: "Quote the premium for an excess",
  inputSchema: z.object({ excess: z.number() }),
  execute: ({ excess }) => ({ premium: excess * 2 }),
});

States and events are inferred from the spec, so a misspelled send is a compile error. procedure() is the sibling for a flow YOU drive with no model in the loop — an XState machine with branches, retries and a bounded budget, run with await procedure(machine).run(input).

Understanding mid-turn

Agents constantly need small acts of understanding inside a turn: pull the order out of a rambling sentence, classify an intent, grade a retrieved document. ctx.generate() is one LLM call, host-side, with a guaranteed shape back:

import { tool } from "@alexkroman1/aai";
import { z } from "zod";

export default tool({
  description: "Start an order from what the caller just said",
  inputSchema: z.object({ utterance: z.string() }),
  execute: async ({ utterance }, ctx) => {
    const parsed = await ctx.generate({
      prompt: `Extract the order from: "${utterance}"`,
      schema: z.object({ sku: z.string(), qty: z.number() }),
    });
    // `object` is typed by the schema: { sku: string; qty: number }.
    return parsed.object;
  },
});

It defaults to the agent's own llm; pass an llm descriptor or a model-id string to use any provider whose key is in the agent's secrets — which is also how an S2S agent reaches a model.

Subagents

Complex requests take an unknown number of lookups the caller never needs to hear. Run inline they bloat the conversation and slow every later turn; delegated, the legwork happens off to the side — in parallel when there are several angles — and the call carries only the answer. A subagent is a second tool loop with its own system prompt, model, tools and context window:

import { speaker, tool } from "@alexkroman1/aai";
import { z } from "zod";

const researcher = speaker({
  name: "researcher",
  systemPrompt:
    "Research the task with the tools you have. IMPORTANT: your final message " +
    "is the only thing the caller sees — end with a self-contained summary.",
  builtinTools: ["web_search", "visit_webpage"],
  maxSteps: 6,
});

export default tool({
  description: "Research a question in depth",
  inputSchema: z.object({ question: z.string() }),
  execute: async ({ question }, ctx) => {
    const { text, toolCalls } = await ctx.delegate(researcher, { task: question });
    return { answer: text, lookups: toolCalls.length };
  },
});

You receive its FINAL message, and its context is isolated — so tell it to summarize, and write task as a complete brief. Delegation is one level deep.

Work that outlives the call

No caller holds four minutes while a report generates. A tool starts a durable run and answers the turn; the run is journaled, so it survives a restart, a redeploy and an idle sandbox. With notify the agent speaks the result when it lands — an ordinary interruptible turn, only if the caller is still on the line. If they've hung up, the run completes anyway and the next call finds it by key.

import { tool, workflow, type WorkflowContext } from "@alexkroman1/aai";
import { z } from "zod";

// workflows/research.ts — an ordinary async function of its input and a ctx.
export async function researchFlow({ topic }: { topic: string }, ctx: WorkflowContext) {
  // `ctx.step` runs once and journals what it returned; a replay never re-runs
  // it. The body itself makes no undurable decision — no Date.now(), no fetch.
  const findings = await ctx.step("gather", () => gather(topic));
  // Suspended, not blocked: the container may exit here and resume when due.
  // Ten minutes and six hours work the same way.
  await ctx.sleep("settle", 10 * 60 * 1000);
  return findings;
}

declare function gather(topic: string): Promise<{ topic: string; points: string[] }>;

export const research = workflow({
  description: "Research a topic and report back",
  input: z.object({ topic: z.string() }),
  run: researchFlow,
});

// tools/research_topic.ts — promise made, and kept.
export default tool({
  description: "Research a topic in the background",
  inputSchema: z.object({ topic: z.string() }),
  execute: async ({ topic }, ctx) => {
    await ctx.workflows.start(research, { topic }, {
      key: ctx.sessionId,
      notify: "Tell the caller what it found.",
    });
    return { started: true };
  },
});

Declare the workflow on the agent (agent({ workflows: { research } })) and ctx.workflows can start, find, get, wakeUp and stream its runs. When the product IS the form rather than the call, workflowApp() is an agent whose front door is a workflow — no session, no LLM loop, and every voice knob a compile error.

Voice in, screen out

The caller talks while a cart, form or itinerary updates on screen — voice for input, screen for confirmation. That is how an order gets placed without reading sixteen digits aloud. syncState takes a PROJECTION, not a flag, so you choose what the browser is allowed to see:

import { agent, sessionSlot } from "@alexkroman1/aai";

// shared.ts — declare the view on the slot and import it at both ends.
// `staffPin` is not in the view, so it never reaches the browser.
export const cartSlot = sessionSlot("cart", () => ({ cart: [] as string[], staffPin: "" }), {
  view: (s) => ({ cart: s.cart }),
  sync: true,
});

export default agent({ name: "Storefront" }); // the `sync: true` slot is collected

import { sessionSlot } from "@alexkroman1/aai";
import { mountClient, useAgentState } from "@alexkroman1/aai-ui";

// client.tsx — in a project this is `import { cartSlot } from "./shared.ts"`,
// the same slot the agent declared, so the two ends cannot name different views.
const cartProjection = sessionSlot("cart", () => ({ cart: [] as string[] }), {
  view: (s) => ({ cart: s.cart }),
}).projected;

function App() {
  // The projection types the state AND supplies the frame rendered before the
  // first push — no type argument, no `?? EMPTY`.
  const view = useAgentState(cartProjection);
  return (
    <ul>
      {view.cart.map((item) => (
        <li key={item}>{item}</li>
      ))}
    </ul>
  );
}

mountClient({ component: App });

The projection runs after every tool call and is sent only when the result changed. ctx.send(event, data) is the push channel for everything that isn't state; @alexkroman1/aai-ui has the hooks and components.

One agent, every line

Name the carriers your number is with, and the agent answers calls on the same session it answers a browser with:

import { agent } from "@alexkroman1/aai";

export default agent({
  name: "Support",
  // `true` admits every carrier this build decodes; omit it and the phone
  // route is not served at all.
  telephony: ["twilio"],
});

Then point the number's voice webhook at the agent and answer with a media stream:

wss://<your-agent-url>/phone?carrier=twilio
wss://<your-agent-url>/phone?carrier=telnyx

Twilio and Telnyx are the carriers this build decodes; both speak 8 kHz mu-law, which the bridge transcodes in both directions, so the same turn-taking, barge-in, tools and slots run on a call as in the browser. A phone call is a transport, not a mode — what you tested in the web demo is what callers get.

Session modes and providers

Pipeline mode (default) streams STT partials into a server-side LLM loop and speaks the reply through a TTS provider. Swap any stage with a factory from the provider subpaths — set any subset of stt, llm, tts; the unset stages keep the AssemblyAI default:

Factories return pure descriptors — serializable data, not SDK clients. Credentials are resolved server-side from the agent's env (each factory's docs name the env var), so no provider SDK or secret ever enters the agent bundle. llm also accepts a model-id string: "creator/model" routes through the Vercel AI Gateway, a bare id through the AssemblyAI LLM Gateway — agent({ name: "...", llm: "claude-sonnet-4-6" }) swaps just the model.

S2S mode is the explicit opt-in to a speech-to-speech service, where STT, the LLM loop, and TTS all run service-side over one socket: s2s: assemblyAIS2s() (root export) or openAIS2s() from @alexkroman1/aai/s2s.

Testing a tool

A complex agent is a living codebase — dozens of tools, an evolving prompt, several people editing it. A tool is a file, so agent.ts's default export carries no tools; deployedAgent gives you the definition a deployed agent runs, and each tool is then callable in isolation with no session, no model and no network:

// file: agent.ts
import { agent, sessionSlot } from "@alexkroman1/aai";

export const notesSlot = sessionSlot("notes", () => ({ items: [] as string[] }));

export default agent({ name: "Notes" });

// file: tools/add_note.ts
import { z } from "zod";
import { notesSlot } from "../agent.ts";

export default notesSlot.updateTool({
  description: "Save a note for the caller",
  inputSchema: z.object({ text: z.string() }),
  execute: ({ text }, notes) => {
    notes.items.push(text);
    return { saved: notes.items.length };
  },
});

// file: agent.test.ts
import { createToolContext, deployedAgent, toolRunner } from "@alexkroman1/aai-runtime/testing";
import { expect, test } from "vitest";
import authored from "./agent.ts";
import addNote from "./tools/add_note.ts";

const agentDef = deployedAgent(authored, {
  tools: import.meta.glob("./tools/*.ts", { eager: true }),
});
const run = toolRunner(agentDef);

test("saves a note", async () => {
  // Typed by the tool's own def: `saved` is a number, no cast.
  const { saved } = await run(addNote, { text: "milk" }, createToolContext());
  expect(saved).toBe(1);
});

createToolContext() builds a full ToolContext with inert defaults and a recording ctx.send; stubGenerate, stubDelegate, stubGateway and stubUploads drive what a tool's collaborators answer. aai build runs the suite and type-checks the project before it bundles.

Behavioral evals

A prompt tweak that improves tone can quietly degrade tool discipline, and nobody hand-tests forty scenarios per edit. describeEval (from @alexkroman1/aai-runtime/testing/vitest, run by aai eval) drives a real session and asserts what the agent DID — which tools it called, in what order, and what it said:

import { agent } from "@alexkroman1/aai";
import { describeEval } from "@alexkroman1/aai-runtime/testing/vitest";
import { expect } from "vitest";

const agentDef = agent({ name: "Support" }); // the prompt is system-prompt.md

describeEval(agentDef, (test) => {
  test(
    "looks the order up first",
    async ({ session }) => {
      const turn = await session.say("where is order W1234?");
      expect(turn.toolCalls.map((c) => c.name)).toContain("look_up");
      expect(turn.text).toMatch(/shipped/i);
    },
    // What a scripted model answers when there is no provider key — the eval
    // still proves the agent boots, tools resolve, and a reply comes back.
    { stubReply: "Order W1234 shipped yesterday." },
  );
});

With a provider key the same file drives a live model; without one it runs scripted, so it is free and deterministic in CI.

Other subpaths

Each subpath is named by WHO READS IT — reach for one when the right-hand column describes what you are doing.

Documentation

Full API reference: https://alexkroman.github.io/agent/

The complete authoring guide — every agent() field, the workflow rules, the UI hooks, the voice-prompt rules — ships in this package as AGENT_GUIDE.md, beside this file. (Deliberately not a relative link: TypeDoc copies a linked file into the rendered docs tree, and this one is 100 kB of generated text that already lives in two places.)