@inferencesh/sdk — ai inference api for javascript & typescript

official javascript/typescript sdk for inference.sh — the ai agent runtime for serverless ai inference.

run ai models, build ai agents, and deploy generative ai applications with a simple api. access models including flux, stable diffusion, llms (claude, gpt, gemini), video generation (veo, seedance), and more.

Installation

npm install @inferencesh/sdk
# or
yarn add @inferencesh/sdk
# or
pnpm add @inferencesh/sdk

Getting an API Key

Get your API key from the inference.sh dashboard.

Quick Start

import { inference, TaskStatusCompleted } from '@inferencesh/sdk';

const client = inference({ apiKey: 'your-api-key' });

// Run a task and wait for the result
const result = await client.tasks.run({
  app: 'your-app',
  input: {
    prompt: 'Hello, world!'
  }
});

if (result.status === TaskStatusCompleted) {
  console.log(result.output);
}

Usage

Basic Usage

import { inference, TaskStatusCompleted } from '@inferencesh/sdk';

const client = inference({ apiKey: 'your-api-key' });

// Wait for result (default behavior)
const result = await client.tasks.run({
  app: 'my-app',
  input: { prompt: 'Generate something amazing' }
});

if (result.status === TaskStatusCompleted) {
  console.log('Output:', result.output);
}

With Setup Parameters

Setup parameters configure the app instance (e.g., model selection). Workers with matching setup are "warm" and skip setup:

const result = await client.tasks.run({
  app: 'my-app',
  setup: { model: 'schnell' },  // Setup parameters
  input: { prompt: 'hello' }
});

Fire and Forget

// Get task info immediately without waiting
const task = await client.tasks.run(
  { app: 'my-app', input: { prompt: 'hello' } },
  { wait: false }
);

console.log('Task ID:', task.id);
console.log('Status:', task.status);

Real-time Status Updates

By default, the client streams task progress over NDJSON (/tasks/{id}/stream) and invokes onUpdate as the task changes. Use onPartialUpdate when you only need specific fields from a partial stream payload:

const result = await client.run(
  { app: 'my-app', input: { prompt: 'hello' } },
  {
    onUpdate: (update) => {
      console.log('Status:', update.status);
      console.log('Progress:', update.logs);
    },
    onPartialUpdate: (update, fields) => {
      console.log('Changed fields:', fields, update.status);
    },
  }
);

Streaming vs Polling

SSE/NDJSON streaming is the default. For edge runtimes that cannot keep long-lived connections open (Convex actions, Cloudflare Workers, etc.), disable streaming and use lightweight status polling instead:

const client = inference({
  apiKey: 'your-api-key',
  stream: false,           // poll /tasks/{id}/status instead of streaming
  pollIntervalMs: 2000,    // default: 2000
});

// Per-call override
const result = await client.run(
  { app: 'my-app', input: { prompt: 'hello' } },
  { stream: false, onUpdate: (u) => console.log(u.status) }
);

In polling mode, the SDK checks /tasks/{id}/status and fetches the full task when the status changes. If that fetch fails after a status transition, run() rejects with the underlying error.

Batch Processing

async function processImages(images: string[]) {
  const results = [];

  for (const image of images) {
    const result = await client.tasks.run({
      app: 'image-processor',
      input: { image }
    }, {
      onUpdate: (update) => console.log(`Processing: ${update.status}`)
    });

    results.push(result);
  }

  return results;
}

File Upload

// Upload from base64
const file = await client.files.upload('data:image/png;base64,...', {
  filename: 'image.png',
  contentType: 'image/png'
});

// Use the uploaded file in a task
const result = await client.tasks.run({
  app: 'image-app',
  input: { image: file.uri }
});

Cancel a Task

const task = await client.tasks.run(
  { app: 'long-running-app', input: {} },
  { wait: false }
);

// Cancel if needed
await client.tasks.cancel(task.id);

Stream Functions (Live Sockets)

A stream function keeps a socket open with its caller for the life of the task: frames go both ways until the caller closes or the app returns. client.live starts the task and dials its socket; the run response carries where to dial (task.socket).

const { task, session } = await client.live(
  { app: 'infsh/voice-loop', function: 'stream', input: { effect: 'robot' } },
  {
    onState: (state) => console.log(state), // connecting → waiting → live → ended
    onBinary: (pcm) => speaker.write(new Int16Array(pcm)),
    onPatch: (patch) => console.log(patch), // e.g. { frames: 120 }
  }
);

session.sendBinary(micFrame);          // one item of the input's binary live field
session.sendPatch({ effect: 'echo' }); // change an ordinary input while it runs
session.close();                       // the function returns and the task completes
await session.ended;

The session is waiting until the app's first frame (a cold start can take a minute) and gives up if the task ends before then. It dials again with a fresh credential when the relay restarts under it. client.sockets.open(taskOrId, handlers) reconnects to a running task's socket, e.g. after a page reload.

What a function's socket carries is in its schemas: a live field is {"type": "array", "format": "stream", "items": ...}. splitLiveSchema(schema) separates the ordinary fields (the request body) from the live ones, and pcmFormat(field.media) reads the sample rate of a PCM audio field.

On Node 18–21 there is no global WebSocket: pass one from the ws package as { webSocket: WebSocket }.

Sessions (Stateful Execution)

Sessions allow you to maintain state across multiple task invocations. The worker stays warm between calls, preserving loaded models and in-memory state.

// Start a new session
const result = await client.tasks.run({
  app: 'my-stateful-app',
  input: { prompt: 'hello' },
  session: 'new'
});

const sessionId = result.session_id;
console.log('Session ID:', sessionId);

// Continue the session with another call
const result2 = await client.tasks.run({
  app: 'my-stateful-app',
  input: { prompt: 'remember what I said?' },
  session: sessionId
});

Custom Session Timeout

By default, sessions expire after 60 seconds of inactivity. You can customize this with session_timeout (1-3600 seconds):

// Create a session with 5-minute idle timeout
const result = await client.tasks.run({
  app: 'my-stateful-app',
  input: { prompt: 'hello' },
  session: 'new',
  session_timeout: 300  // 5 minutes
});

// Session stays alive for 5 minutes after each call

Notes:

• session_timeout is only valid when session: 'new'
• Minimum timeout: 1 second
• Maximum timeout: 3600 seconds (1 hour)
• Each successful call resets the idle timer

Session management API

Manage sessions directly without running a task:

// Inspect a session
const info = await client.sessions.get(sessionId);
console.log(info.status, info.expires_at, info.call_count);

// List active sessions
const sessions = await client.sessions.list();

// Extend idle timeout without a task call (sliding window)
await client.sessions.keepalive(sessionId);

// Release the worker immediately
await client.sessions.end(sessionId);

Session errors

import {
  SessionNotFoundError,
  SessionExpiredError,
  SessionEndedError,
} from '@inferencesh/sdk';

try {
  await client.tasks.run({
    app: 'my-stateful-app',
    input: { prompt: 'hello' },
    session: sessionId,
  });
} catch (error) {
  if (
    error instanceof SessionNotFoundError ||
    error instanceof SessionExpiredError ||
    error instanceof SessionEndedError
  ) {
    // Start a new session and retry
    const result = await client.tasks.run({
      app: 'my-stateful-app',
      input: { prompt: 'hello' },
      session: 'new',
    });
  } else {
    throw error;
  }
}

For complete session documentation including error handling, best practices, and advanced patterns, see the Sessions Developer Guide.

Agent Chat

Chat with AI agents using client.agents.create().

Using a Template Agent

Use an existing agent from your workspace by its namespace/name@shortid:

import { inference } from '@inferencesh/sdk';

const client = inference({ apiKey: 'your-api-key' });

// Create agent from template
const agent = client.agents.create('my-org/assistant@abc123');

// Send a message with streaming
await agent.sendMessage('Hello!', {
  onMessage: (msg) => {
    if (msg.content) {
      for (const c of msg.content) {
        if (c.type === 'text' && c.text) {
          process.stdout.write(c.text);
        }
      }
    }
  }
});

// Clean up
agent.disconnect();

Creating an Ad-Hoc Agent

Create agents on-the-fly without saving to your workspace:

import { inference, tool, string } from '@inferencesh/sdk';

const client = inference({ apiKey: 'your-api-key' });

// Create ad-hoc agent (config uses API field names: core_app, system_prompt)
const weatherTool = tool('get_weather')
  .describe('Get current weather')
  .param('city', string('City name'))
  .build();

const agent = client.agents.create({
  core_app: { ref: 'infsh/claude-sonnet-4@abc123' },
  system_prompt: 'You are a helpful assistant.',
  tools: [weatherTool], // only schemas are sent to the API; handlers stay client-side
});

await agent.sendMessage('What is the weather in Paris?', {
  onMessage: (msg) => console.log(msg),
  onToolCall: async (call) => {
    const result = await runMyClientTool(call.name, call.args);
    await agent.submitToolResult(call.id, result);
  },
});

For multi-turn chats, the SDK opens the chat stream before sending the next message so updates are not missed. Use stopChat() to cancel in-flight generation (POST /chats/{id}/stop), and reset() to clear the current chat and start fresh.

Tool builder

Use the fluent builders to define AgentTool schemas. Client tools (tool) run in your app via onToolCall; server-side tools run on inference.sh.

import {
  inference,
  tool,
  appTool,
  httpTool,
  mcpTool,
  internalTools,
  string,
  CredentialProviderGoogle,
} from '@inferencesh/sdk';

const clientTool = tool('get_weather')
  .describe('Get current weather')
  .param('city', string('City name'))
  .build();

// HTTP tool authenticated with a connected account (token injected server-side)
const gmailSend = httpTool('gmail_send', 'https://gmail.googleapis.com/gmail/v1/users/me/messages/send')
  .describe('Send an email via Gmail')
  .method('POST')
  .auth({ credential: CredentialProviderGoogle })
  .build();

// API key or bearer auth
const fetchData = httpTool('fetch', 'https://api.example.com/data')
  .method('GET')
  .auth({ apiKey: 'YOUR_KEY', header: 'X-API-Key' }) // default header: X-API-Key
  .header('Accept', 'application/json')
  .build();

const bearerFetch = httpTool('bearer_fetch', 'https://api.example.com')
  .auth({ bearer: 'YOUR_TOKEN' })
  .build();

const imageGen = appTool('generate_image', 'infsh/flux-schnell@abc123')
  .param('prompt', string('Image description'))
  .requireApproval()
  .build();

// MCP server by slug; the runner's connection to its provider signs the call
const mcpSearch = mcpTool('notion_search', { server: 'notion', toolName: 'search' })
  .describe('Search Notion pages')
  .param('query', string('Search query'))
  .build();

const agent = client.agents.create({
  core_app: { ref: 'infsh/claude-sonnet-4@latest' },
  system_prompt: 'You are helpful.',
  tools: [clientTool, gmailSend, imageGen, mcpSearch],
  internal_tools: internalTools().memory().build(),
});

callTool is an alias for httpTool. Run npx tsx examples/tool-builder.ts for more schema examples (no API key required).

Lifecycle hooks

Use lifecycleHook() to attach hooks to agent events. Hooks can call a webhook or delegate to another agent as a task:

import { inference, lifecycleHook, HookEventTurnStart, HookEventToolCall } from '@inferencesh/sdk';

const client = inference({ apiKey: 'your-api-key' });

const agent = client.agents.create({
  core_app: { ref: 'openrouter/claude-sonnet-4@abc' },
  hooks: [
    lifecycleHook(HookEventTurnStart)
      .webhook('https://example.com/on-turn')
      .timeout(10)
      .build(),
    lifecycleHook(HookEventToolCall)
      .task('my-org/approval-agent')
      .async(true)
      .build(),
  ],
});

Queued messages

sendMessage automatically queues when the agent is busy processing a previous message. Queued messages appear with a queued status until the agent picks them up.

To cancel a queued message before it is processed, use cancelMessage on the React provider or chats.cancelMessage on the API client:

// Via the API client
await client.chats.cancelMessage(messageId);

File attachments

Pass files in sendMessage options. Blob values are uploaded first; objects with a uri (already uploaded via client.files.upload) are attached as-is:

const uploaded = await client.files.upload(imageBlob, {
  filename: 'photo.png',
  contentType: 'image/png',
});

await agent.sendMessage('Describe this image', {
  files: [imageBlob, uploaded], // Blob uploads; FileDTO reuses uri
  onMessage: (msg) => console.log(msg),
});

Structured Output

Use output_schema to get structured JSON responses:

const agent = client.agents.create({
  core_app: { ref: 'infsh/claude-sonnet-4@latest' },
  output_schema: {
    type: 'object',
    properties: {
      summary: { type: 'string' },
      sentiment: { type: 'string', enum: ['positive', 'negative', 'neutral'] },
      confidence: { type: 'number' },
    },
    required: ['summary', 'sentiment', 'confidence'],
  },
  internal_tools: { finish: true },
});

const output = await agent.run('Analyze: Great product!');

agent.run() sends a message with polling (no SSE), waits until the chat is idle, and returns chat.output (parsed finish-tool result, or null if none).

Agent Methods

Messages that arrive through a channel integration (Slack, Telegram, a wearable) can carry channel_context so the reply is routed back the same way. The API records it on the chat the first time it sees it, so later turns inherit it:

await agent.sendMessage(text, {
  channel_context: { channel_type: 'slack', channel_metadata: { channel: 'C123', thread_ts: '1700000000.000100' } },
});

API Reference

inference(config)

Creates a new inference client.

client.tasks.run(params, options?)

Runs a task on inference.sh.

Parameters:

Options:

client.tasks.get(taskId)

Gets a task by ID.

client.tasks.cancel(taskId)

Cancels a running task.

client.files.upload(data, options?)

Uploads a file to inference.sh.

Parameters:

client.agents.create(templateOrConfig) / client.agent(...)

Creates an agent instance from a template or ad-hoc configuration. client.agent(...) is an alias for client.agents.create(...).

sendMessage options: onMessage, onChat, onToolCall, files, stream, pollIntervalMs. Client tools with status awaiting_input are dispatched once per invocation ID via onToolCall.

Template mode:

const agent = client.agents.create('namespace/name@version');

Ad-hoc mode:

const agent = client.agents.create({
  core_app: { ref: 'infsh/claude-sonnet-4@abc123' },
  system_prompt: 'You are helpful.',
  tools: [...],
});

client.sessions

client.sockets

client.live(params, handlers?, options?) is tasks.run(params, { wait: false }) followed by sockets.open.

Task Status Constants

import {
  TaskStatusQueued,
  TaskStatusRunning,
  TaskStatusCompleted,
  TaskStatusFailed,
  TaskStatusCancelled
} from '@inferencesh/sdk';

if (task.status === TaskStatusCompleted) {
  console.log('Done!');
}

Credential Constants

CredentialDTO fields (provider, type, status) use typed string unions exported as constants:

import type { CredentialDTO } from '@inferencesh/sdk';
import {
  CredentialProviderGoogle,
  CredentialTypeOAuth,
  CredentialStatusConnected,
  CredentialStatusDisconnected,
  CredentialStatusExpired,
  CredentialStatusError,
  RequirementTypeCredential,
  isRequirementsNotMetException,
} from '@inferencesh/sdk';

function isGoogleConnected(credential: CredentialDTO): boolean {
  return (
    credential.provider === CredentialProviderGoogle &&
    credential.status === CredentialStatusConnected
  );
}

// HTTP 412 when an app requires a missing secret, credential, or scope
try {
  await client.run({ app: 'my-app', input: {} });
} catch (error) {
  if (isRequirementsNotMetException(error)) {
    for (const req of error.errors) {
      if (req.type === RequirementTypeCredential && req.action?.provider === CredentialProviderGoogle) {
        // User must connect Google — see https://inference.sh/docs/extend/credentials
      }
    }
  }
}

Instance Status Constants

Use when working with engine instance APIs (InstanceDTO.status):

import {
  InstanceStatusCreating,
  InstanceStatusPendingProvider,
  InstanceStatusPending,
  InstanceStatusActive,
  InstanceStatusError,
  InstanceStatusDeleting,
  InstanceStatusDeleted,
} from '@inferencesh/sdk';

Tool Parameter Types

When building AgentTool schemas manually (outside the tool builder), use ToolParamType* for JSON Schema type fields:

import {
  ToolParamTypeObject,
  ToolParamTypeString,
  ToolParamTypeInteger,
  ToolParamTypeNumber,
  ToolParamTypeBoolean,
  ToolParamTypeArray,
  ToolParamTypeNull,
} from '@inferencesh/sdk';

const schema = {
  type: ToolParamTypeObject,
  properties: {
    city: { type: ToolParamTypeString, description: 'City name' },
  },
  required: ['city'],
};

The fluent tool builder (string(), number(), object(), …) infers these types automatically.

TypeScript Support

This SDK is written in TypeScript and includes full type definitions. All types are exported:

import type {
  Task,
  ApiAppRunRequest,
  RunOptions,
  CredentialDTO,
  AgentTool,
} from '@inferencesh/sdk';

Requirements

• Node.js 18.0.0 or higher
• Modern browsers with fetch support

resources

• documentation — getting started guides and api reference
• blog — tutorials on ai agents, image generation, and more
• app store — browse ai models
• discord — community support
• github — open source projects

license

MIT © inference.sh