@ai-setu/client

Read this first, agent. This package = drop-in for the official openai SDK, pointed at the AI Setu gateway. If you know OpenAI SDK, you know this. Same methods, same shapes, same streaming. Only thing change: one workspace key, and model carry a provider/ prefix. Everything below is copy-paste safe.

TL;DR (the only three lines you need)

npm i @ai-setu/client
export AI_SETU_API_KEY=tt_live_…

import { AiSetu } from '@ai-setu/client';
const client = new AiSetu();
const res = await client.chat.completions.create({
  model: 'openai/gpt-4o-mini',
  messages: [{ role: 'user', content: 'hi' }],
});
console.log(res.choices[0].message.content);

Done. new AiSetu() with no args read AI_SETU_API_KEY from env. No baseURL, no createClient, no factory. One import, one symbol.

Rules for agent (follow these, no guess)

1. Model string provider/model (platform key) or @<slug>/<model> (your BYOK Connection — see below). Good: openai/gpt-4o-mini, anthropic/claude-sonnet-4-5, @azure-qdc/gpt-4o-mini. A bare name with no prefix routes by catalog.
2. No pick endpoint. Client talk to gateway only. You no set URLs for normal use. Provider chosen by the model prefix, not by base URL.
3. Method names = OpenAI SDK exactly. chat.completions.create, embeddings.create, streaming by stream: true + for await. Whatever you learned on openai, it transfer here unchanged.
4. Key go in env AI_SETU_API_KEY (shape tt_live_…). Or pass new AiSetu({ apiKey }). Never hardcode key in source you commit.
5. Need balance / usage / keys? That NOT this package. Use @ai-setu/admin. This one stay pure inference.
6. Node ≥ 20 or edge runtime (Vercel, Cloudflare Workers).

Models — pick by prefix

openai/gpt-4o            openai/gpt-4o-mini       openai/text-embedding-3-small
anthropic/claude-sonnet-4-5                       anthropic/claude-haiku-4-5
bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0   (BYOK — your AWS account)

Streaming (same as OpenAI)

const stream = await client.chat.completions.create({
  model: 'anthropic/claude-sonnet-4-5',
  messages: [{ role: 'user', content: 'Write me a haiku.' }],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? '');
}

Responses API (client.responses)

Same SDK method you already know — it now reach the gateway on /v1/responses.

const r = await client.responses.create({
  model: 'openai/gpt-4o-mini',
  instructions: 'You are a concise support assistant.',
  input: [{ role: 'user', content: 'do you have anything in a darker fabric' }],
  max_output_tokens: 400,
});

Two rules, both easy to trip:

1. OpenAI-family models only. openai, bedrock-mantle, groq, xai, deepseek, mistral. Anthropic, Gemini, Bedrock and Vertex have no Responses surface — they fail closed with a clear error, no silent fallback.
2. Reasoning spell different per surface. Here it reasoning: { effort: 'none' }. On chat.completions it reasoning_effort: 'none'. Each reject the other.

Usage come back in the Responses shape — usage.input_tokens / output_tokens, cache counters under usage.input_tokens_details — not the chat shape. promptCacheStats read both, so no branch needed.

Semantic caching no apply on this route. It and per-tool cache policies read messages, which a Responses body no carry. Exact caching, prompt caching, budgets, rate limits all work normal. Need semantic? Use chat.completions.

What the gateway actually route

AiSetu extend the openai SDK, so your editor show every OpenAI method. Inherited is not the same as routed:

Embeddings

const e = await client.embeddings.create({
  model: 'openai/text-embedding-3-small',
  input: 'hello world',
});

Errors — catch logic from OpenAI SDK transfer straight

Use type guards, no instanceof on subclasses. Reason: the openai SDK make APIError itself from inside its pipeline, so subclass drift every release. Guard narrow the type back to APIError — you still get err.code, err.status, err.requestID.

import {
  AuthenticationError,
  RateLimitError,
  isInsufficientCreditsError,
  getInsufficientCreditsDetails,
  isProviderError,
} from '@ai-setu/client';

try {
  await client.chat.completions.create({
    /* … */
  });
} catch (err) {
  if (err instanceof AuthenticationError) {
    /* bad key */
  } else if (err instanceof RateLimitError) {
    /* slow down */
  } else if (isInsufficientCreditsError(err)) {
    const { balanceMicros, topupUrl } = getInsufficientCreditsDetails(err);
    console.error(`Out of credits (balance ${balanceMicros} micros). Top up: ${topupUrl}`);
  } else if (isProviderError(err)) {
    /* upstream provider broke */
  }
}

Environments — one switch move whole app

Default = production (gateway.aisetu.ai). Point at a different gateway (e.g. a self-hosted deployment) with a full URL override:

export AI_SETU_BASE_URL=https://gateway.your-deployment.example.com

Precedence (first match win):

1. new AiSetu({ baseUrl }) — explicit per-instance
2. AI_SETU_BASE_URL — full URL override (self-host / on-prem)
3. production default

(Base always normalise to end in /v1.)

Runtime + connection pooling (you usually no touch this)

• Node ≥ 20 — one process-wide keep-alive undici.Agent back every AiSetu. Sequential calls reuse the TCP connection (~5ms p50 vs ~30ms cold).
• Edge (Cloudflare Workers / Vercel) — fall back to platform fetch, which already pool. undici is optional dep, edge bundlers skip it.

Pool defaults: connections 32, allowH2 true, keepAliveTimeout 30s, keepAliveMaxTimeout 90s (match gateway idle), bodyTimeout 0 (no cap — long thinking-model streams can run for minutes; cancel with AbortSignal).

Override the pool only if you must:

import { Agent, ProxyAgent } from 'undici';
import { AiSetu } from '@ai-setu/client';

// Corporate proxy:
const proxied = new AiSetu({ dispatcher: new ProxyAgent({ uri: 'http://proxy:8080' }) });

// Self-signed CA (baseUrl normalise to …/v1):
const onPrem = new AiSetu({
  baseUrl: 'https://gateway.internal.example',
  dispatcher: new Agent({ connect: { ca: fs.readFileSync('ca.pem') } }),
});

// No pooling at all (platform fetch on Node):
const unpooled = new AiSetu({ dispatcher: null });

Process-wide swap (before any AiSetu construct), or share the singleton with sibling SDKs:

import { setSharedDispatcher, getSharedDispatcher } from '@ai-setu/client';
import { Agent } from 'undici';
import { AiSetuAdmin } from '@ai-setu/admin';

setSharedDispatcher(new Agent({ connections: 64, allowH2: true }));
const admin = new AiSetuAdmin({ dispatcher: getSharedDispatcher() });

BYOK: route per request with @<slug>/<model>

Your provider keys live in Connections (each has a slug). Pick one per request inside the model string — @<slug>/<model> — and the gateway routes through that Connection's key. One client, mix Connections freely, no re-init. A leading @ marks a slug (so a slug can equal a provider name); an unknown or out-of-scope slug fails closed (4xx), never a silent platform-key fallback.

const c = new AiSetu();

await c.chat.completions.create({
  model: '@azure-qdc/gpt-5.4-mini', // route via the Connection slug "azure-qdc"
  messages: [{ role: 'user', content: 'hi' }],
});

await c.chat.completions.create({
  model: 'openai/gpt-4o-mini', // platform key (no @slug)
  messages: [{ role: 'user', content: 'hi' }],
});

Pin one Connection for the whole client with the connection option (a leading @ is optional):

const c = new AiSetu({ connection: '@azure-qdc' });

Caching — turn it on in the SDK, not in headers

Cheapest saving first, in the order worth adopting.

1. Prompt caching — already on, costs you nothing

The provider skips re-prefilling the unchanged prefix of your prompt (system prompt, tool definitions, settled conversation) and discounts those tokens. The model still runs, the output is still fresh — no staleness risk at all. Nothing to switch on. Most agent workloads get half their prompt discounted this way without noticing:

import { promptCacheStats } from '@ai-setu/client';

const res = await c.chat.completions.create({ model, messages });
promptCacheStats(res);
// { inputTokens: 1000, cachedTokens: 3000, writeTokens: 0, ratio: 0.75 }

ratio is the share of the prompt that avoided a full-price prefill. Reads both the OpenAI and Anthropic usage shapes, so you don't branch on provider. Keep the stable part of your prompt first and unchanged — a timestamp or a session id at the top of the system prompt destroys the prefix for every call.

2. Response caching — a hit costs $0 and returns in ~3ms

Default: the gateway caches when temperature is unset or 0, keyed on the exact request body, for 600 seconds. Set client-wide defaults, then override per request:

const c = new AiSetu({
  cache: {
    policy: 'support-faq', // TTL + stale window live in the console, not in code
    namespace: 'tenant-42', // keep one end-user's entries off another's
  },
});

await c.chat.completions.create(
  { model: '@mantle/zai.glm-4.7-flash', messages },
  c.withCache({ ttl: '2h', tags: ['sku:123'] }),
);

A policy plus tags beats a long bare TTL: tags let you hold an answer for hours and still drop it the moment the underlying data changes.

Bad values throw — a typo'd policy name would otherwise be dropped by the gateway and leave you with caching that silently does nothing.

3. Caching inside agent loops — policy per tool

Agent turns are the most expensive traffic and the least cacheable, because a tool result embeds live data. You can't decide that per request — incoming text is arbitrary. But you do know what each tool does when you add it, so the policy rides with the tool:

import { withToolCache } from '@ai-setu/client';

const tools = [
  withToolCache(getCatalog, { policy: 'catalog', tags: ['catalog', 'sku:{sku}'] }),
  withToolCache(getBrandStory, { policy: 'brand-voice' }),
  withToolCache(addToCart, { cache: false }),
  withToolCache(checkout, { cache: false }),
];

The turn's cacheability is derived from the tools it actually invoked. Any tool saying cache: false bypasses the whole turn; TTL and SWR take the minimum across participating tools, so a turn touching the catalogue (60s) and the brand story (24h) is only as fresh as the catalogue. Undeclared tools block nothing.

Bypass is visible rather than silent:

client.lastCache;
// { status: 'bypass', hit: false, semantic: false, bypassedByTool: 'addToCart' }

4. Semantic caching — opt-in, and not the first thing to reach for

mode: 'semantic' matches paraphrases, not just byte-identical bodies. Real value for FAQ-shaped traffic — but it is not free and it is not the default fix for a low hit rate:

• Every exact-miss pays an embedding round trip (~342ms today) before the provider is even called. On a workload whose misses already take ~600ms, that is a large latency add on the common path.
• Match quality is English-leaning. Hindi and Arabic-family paraphrase matching is currently weak, Hinglish middling — code-switching traffic may miss real paraphrases and match across intents.

Use it where answers are short, repetitive, English, and latency-tolerant. Raise threshold (0–1, default 0.8) if near-misses are being served. Exhaust prompt caching, a policy, and tags first.

5. Conversational AI — split the turn, or nothing hits

If you build a voice or chat agent, this is the section that decides your hit rate.

One user utterance normally fans out into several provider calls, and they are not equally cacheable. The reply is personalised and unique per customer per turn. The intent classification behind it depends on nothing but the words the customer said. Ask one prompt to do both and the whole turn inherits the cacheability of the reply — which is zero. That reads as a threshold to tune; it is a decomposition to do.

Issue the shareable parts as their own calls:

import { AiSetu, normalizeUtterance, sharedCacheOptions } from '@ai-setu/client';

// Intent classification — shared by every customer who ever says it.
const classifier = new AiSetu().withCache(sharedCacheOptions({ policy: 'intent' }));
await classifier.chat.completions.create({
  model,
  temperature: 0,
  messages: [
    { role: 'system', content: INTENT_LABELS }, // frozen — interpolate nothing
    { role: 'user', content: normalizeUtterance(transcript) },
  ],
});

// FAQ answer — paraphrase-matched, dropped when the article changes.
const faq = new AiSetu().withCache(
  sharedCacheOptions({ mode: 'semantic', policy: 'faq', tags: ['kb:returns-policy'] }),
);

normalizeUtterance matters more than it looks: the exact cache canonicalises JSON but never rewrites message text, so "Cancel my order" and "cancel my order" are two entries.

sharedCacheOptions sets force (classifiers usually run at temperature > 0) and throws if you pass a namespace. That is deliberate — a namespace partitions the keyspace, so a per-customer one on a shareable call gives you a zero hit rate with no error anywhere. It is the most expensive mistake on this surface.

The highest-leverage trick needs no SDK at all: cache the impersonal body, splice the name in afterwards. If the model does not need to reason over a personal field, keep it out of the prompt.

Full patterns, including the prompt shape that keeps prompt caching working across turns: docs.aisetu.ai/concepts/conversational-caching.

Prove it hit

const c = new AiSetu({
  onCache: (info) => metrics.increment(info.hit ? 'cache.hit' : 'cache.miss'),
});

await c.chat.completions.create({ model, messages });
c.lastCache;
// { status: 'hit', hit: true, semantic: false, ageSeconds: 42 }

CacheInfo:

lastCache stays undefined when the request was not cacheable at all — a temperature > 0 call without force, or mode: 'off'. That is the signal that nothing was even looked up, distinct from a miss.

One hit-rate killer the SDK can't fix for you: spell the model the same way every time. zai.glm-4.7-flash and @mantle/zai.glm-4.7-flash are different cache partitions.

Verify your BYOK Connection served (not the platform key)

The gateway stamps attribution headers on every response; the SDK parses them so you can confirm — per request — that your own key served:

const c = new AiSetu({
  onRouting: (r) => {
    if (!r.byok) throw new Error('expected BYOK, got platform key — bad slug?');
  },
});

await c.chat.completions.create({
  model: '@azure-qdc/gpt-5.4-mini',
  messages: [{ role: 'user', content: 'hi' }],
});

// Or read synchronously after any call:
c.lastRouting;
// { provider: 'openai', byok: true,
//   credentialId: '…uuid…', connectionSlug: 'azure-qdc' }

RoutingInfo:

Callback is best-effort — an exception inside onRouting is swallowed so a faulty listener can't break inference. Headers absent (gateway older than this feature) → no callback, lastRouting stays undefined. Credential id / slug are not secrets — you own the Connection.

Legacy: the workspaceCredential option + the label/model model string are deprecated. workspaceCredential still works (alias of connection); the bare label/model form does not — use @<slug>/<model>.

Why not raw OpenAI SDK with custom baseURL?

You can. AiSetu just do it for you and add: env-var-first construct (new AiSetu() no args), keep-alive pool tuned to the gateway, and per-request BYOK routing (@<slug>/<model> / the connection option). No care about those? Point openai at the gateway yourself.

The four AI Setu packages — which one you grab

Full agent runbook = llms.txt.

License

MIT.