Genkit Middleware
This package provides a collection of useful middlewares for the Genkit JS SDK to enhance model execution, tool usage, and agentic workflows.
Building with a coding agent? Install the Genkit JS skill first.
npx skills add genkit-ai/skills --skill developing-genkit-jsIt teaches your agent the current Genkit JS APIs and common gotchas. Source, manual install and skills for other languages: genkit-ai/skills.
Installation
npm install @genkit-ai/middleware
# or
pnpm add @genkit-ai/middleware
Available Middlewares
1. Agents Middleware (agents)
Enables sub-agent delegation. For each configured agent the middleware injects a dedicated delegation tool (e.g. delegate_to_researcher) and appends a <sub-agents> block to the system prompt listing the available agents and their descriptions. When the model calls a delegation tool, the middleware resolves the target agent from the registry, runs it via its run() method, and returns the sub-agent's response as the tool result.
Key behaviors:
- Injects one delegation tool per agent, named
<toolPrefix>_<agentName>(default prefix:delegate_to). - Agent descriptions are auto-discovered from the registry (or can be overridden per-agent) and surfaced in the system prompt.
- Sub-agent interrupts and failures are returned as tool responses (not thrown), allowing the orchestrator to self-correct. (Interactive, stateful back-and-forth with an interrupted sub-agent is a future feature.) A turn that ends without an answer (
length,blocked,aborted) is reported as an error naming the reason and the agent's last message, and a success with no final text says so and names its artifact count. - Sub-agent artifacts are merged into the parent session and/or returned inline, controlled by
artifactStrategy. - With
async: true, delegations can run in the background and be collected later (see below). - A delegation to a sub-agent with a
storeleaves ataskIdbehind, and thecontinue_tasktool continues it (see below). - A delegation can carry an optional
namelabel, echoed on its result, on background-task reports, and on continuations.
Options:
| Option | Type | Default | Description |
|---|---|---|---|
agents | (string | { name, description? })[] | — (required) | Agents available for delegation. A string is the agent name; the object form lets you override the description. |
toolPrefix | string | 'delegate_to' | Prefix for generated delegation tool names. Set to '' to use bare agent names. A non-empty prefix also namespaces the shared tools (continue_task and the background-task tools), so two instances in one generate call need distinct prefixes. |
maxDelegations | number | unlimited | Maximum sub-agent delegations allowed per generate call. Prevents runaway delegation loops. |
historyLength | number | 0 | Number of recent conversation messages (user/model only) to forward to sub-agents as context. |
artifactStrategy | 'inline' | 'session' | 'inline' | inline: artifact content is included in the tool result and merged into the parent session. session: artifacts are merged into the parent session only (the tool result lists names only). Pair session with the artifacts middleware. |
async | boolean | false | Enables background delegation: delegation tools accept a background flag that returns a task ID immediately, and the check_background_tasks, wait_for_background_tasks, and abort_background_tasks tools are added. Requires sub-agents defined with a store. |
maxWaitSeconds | number | unbounded | Upper bound on one wait_for_background_tasks call, whatever timeoutSeconds the model asks for; at the bound the wait returns the current statuses with timedOut set. |
import { genkit } from 'genkit';
import { agents } from '@genkit-ai/middleware';
const ai = genkit({ ... });
// Define sub-agents
const researcher = ai.defineAgent({
name: 'researcher',
model: 'gemini-flash-latest',
description: 'Searches the web and summarizes findings.',
system: 'You are a research assistant.',
tools: [webSearchTool],
});
const coder = ai.defineAgent({
name: 'coder',
model: 'gemini-flash-latest',
system: 'You are an expert programmer.',
});
// Main orchestrator agent delegates to sub-agents.
// This injects `delegate_to_researcher` and `delegate_to_coder` tools.
const orchestrator = ai.defineAgent({
name: 'orchestrator',
model: 'gemini-flash-latest',
system: 'Delegate research to the researcher and coding to the coder.',
use: [
agents({ agents: ['researcher', 'coder'] })
]
});
You can customize the tool-name prefix and override descriptions:
use: [
agents({
// Tools become `ask_researcher` and `ask_coder`.
toolPrefix: 'ask',
agents: [
'researcher', // description auto-discovered from the registry
{ name: 'coder', description: 'Writes TypeScript code' }, // explicit override
],
maxDelegations: 5,
historyLength: 4,
}),
];
Set async: true to let the orchestrator keep working while a sub-agent runs. Each delegation tool gains a background flag that returns a task ID instead of waiting, and three shared tools give the orchestrator one control per thing it can do with a launched task: check_background_tasks reads statuses without waiting, wait_for_background_tasks blocks until they settle, and abort_background_tasks stops the ones whose results are no longer needed. The sub-agent is what needs the session store here: background work is tracked by a snapshot, so a sub-agent without a store can only be delegated to synchronously.
import { InMemorySessionStore } from 'genkit/beta';
// The sub-agent needs its own store to be delegated to in the background.
const researcher = ai.defineAgent({
name: 'researcher',
model: googleAI.model('gemini-flash-latest'),
description: 'Researches a topic and summarizes well-sourced findings.',
system: 'You are a thorough research assistant.',
store: new InMemorySessionStore(),
});
const orchestrator = ai.defineAgent({
name: 'orchestrator',
model: googleAI.model('gemini-flash-latest'),
system:
'Delegate research in the background and post an update while it runs.',
use: [agents({ agents: ['researcher'], async: true })],
});
The orchestrator then launches with { "task": "...", "background": true }, posts an update while the sub-agent runs, and collects the result later. Task IDs (<agent>:<snapshotId>) ride in the delegation tool result, so they are recorded in the conversation and a re-instantiated orchestrator can collect them from its history alone. The background-task tools and continue_task accept only the task IDs that this conversation's delegation and continue results carry, so text that reaches the model cannot point them at another conversation's task. wait_for_background_tasks takes an optional timeoutSeconds, so a slow task becomes an interim answer instead of a blocked turn; maxWaitSeconds caps it so a sub-agent that never finishes cannot hold the request open. waitFor: "first" turns the join into a race: the tool returns as soon as any listed task settles while the rest keep running. An abort is safe to call on any task: one that had already finished is left alone and reports its result, so the orchestrator never loses an answer by giving up on it. A live task reports aborting while it winds down and settles as aborted with the progress it saved; wait_for_background_tasks follows it there.
Delegations that settle leave a continuable handle behind. A delegation result or background-task report for a sub-agent with a store carries a taskId, and the shared continue_task tool spends it. A failed or aborted task continues from its last saved progress: with no instructions the committed turn is re-attempted as it stood, and with instructions the retry is steered. A completed task accepts follow-up instructions inside the sub-agent's own session, so pressing on never repeats finished work. With async: true, continue_task also takes background: true and returns a fresh task ID in the same session. An empty background continuation is the one exception to the retry: a detach with no message runs no turn, so the sub-agent gets a short "Continue the task from where it stopped." message instead, and its transcript records it. A sub-agent without a store settles inline, carries no taskId, and is redone by delegating again; when no configured sub-agent has a store, the model is not shown continue_task. A task that stopped on an interrupt cannot be continued. An expired task (its worker died) is fenced with an abort and continued from its parent snapshot, the last one committed before the detach; a background launch that died before committing anything has nothing saved, and the tool says to delegate again.
2. Artifacts Middleware (artifacts)
Gives the model tools to interact with session artifacts and injects an <artifacts> listing into the system prompt each turn. Useful standalone (e.g. a workspace-builder agent that produces files as artifacts) or combined with the agents middleware using artifactStrategy: 'session', so the orchestrator can read artifacts produced by sub-agents.
Tools provided:
read_artifact— reads a named artifact from the session and returns its text content.write_artifact— creates or updates a named artifact (deduplicated by name). Omitted whenreadonly: true.
Options:
| Option | Type | Default | Description |
|---|---|---|---|
readonly | boolean | false | When true, only the read_artifact tool is provided. |
import { genkit } from 'genkit';
import { agents, artifacts } from '@genkit-ai/middleware';
const ai = genkit({ ... });
// Standalone: an agent that creates and reads artifacts.
const builder = ai.defineAgent({
name: 'builder',
model: 'gemini-flash-latest',
system: 'You are a code generator. Use write_artifact to create files.',
use: [artifacts()],
});
// Combined with the agents middleware (session strategy).
const orchestrator = ai.defineAgent({
name: 'orchestrator',
model: 'gemini-flash-latest',
system: 'You coordinate sub-agents and review their work.',
use: [
agents({ agents: ['researcher', 'coder'], artifactStrategy: 'session' }),
artifacts({ readonly: true }), // can read sub-agent artifacts
],
});
3. FileSystem Middleware (filesystem)
Grants the model access to the local filesystem by injecting standard file manipulation tools (list_files, read_file, write_file, search_and_replace). All operations are safely restricted to a specified root directory. Note that write operations require setting allowWriteAccess: true in the middleware configuration.
import { genkit } from 'genkit';
import { filesystem } from '@genkit-ai/middleware';
const ai = genkit({ ... });
const response = await ai.generate({
model: 'gemini-flash-latest',
prompt: 'Create a hello world node app in the workspace',
use: [
filesystem({ rootDirectory: './workspace', allowWriteAccess: true })
]
});
4. Skills Middleware (skills)
Automatically scans a directory for SKILL.md files (and their YAML frontmatter) and injects them into the system prompt. It also provides a use_skill tool the model can use to retrieve more specific skills on demand.
import { genkit } from 'genkit';
import { skills } from '@genkit-ai/middleware';
const ai = genkit({ ... });
const response = await ai.generate({
prompt: 'How do I run tests in this repo?',
use: [
skills({ skillPaths: ['./skills'] })
]
});
5. Tool Approval Middleware (toolApproval)
Restricts execution of tools to an approved list. If the model attempts to call an unapproved tool, it throws a ToolInterruptError allowing you to prompt the user for manual confirmation before resuming.
import { genkit, restartTool } from 'genkit';
import { toolApproval } from '@genkit-ai/middleware';
const ai = genkit({ ... });
// 1. Initial attempt
const response = await ai.generate({
prompt: 'write a file',
tools: [writeFileTool],
use: [
toolApproval({ approved: [] }) // Empty list means call triggers interrupt
]
});
if (response.finishReason === 'interrupted') {
const interrupt = response.interrupts[0];
// 2. Ask user for approval, then recreate the tool request with approval
const approvedPart = restartTool(interrupt, { toolApproved: true });
// 3. Resume execution
const resumedResponse = await ai.generate({
messages: response.messages,
resume: { restart: [approvedPart] },
use: [
toolApproval({ approved: [] })
]
});
}
6. Retry Middleware (retry)
Automatically retries failed model generations on transient error codes (like RESOURCE_EXHAUSTED, UNAVAILABLE) using exponential backoff with jitter.
import { genkit } from 'genkit';
import { retry } from '@genkit-ai/middleware';
const ai = genkit({ ... });
const response = await ai.generate({
model: googleAI.model('gemini-pro-latest'),
prompt: 'Heavy reasoning task...',
use: [
retry({
maxRetries: 3,
initialDelayMs: 1000,
backoffFactor: 2
})
]
});
7. Fallback Middleware (fallback)
Automatically switches to a different model if the primary model fails on a specific set of error codes. Useful for falling back to a smaller/faster model when a large model exceeds quota limits.
import { genkit } from 'genkit';
import { fallback } from '@genkit-ai/middleware';
const ai = genkit({ ... });
const response = await ai.generate({
model: googleAI.model('gemini-pro-latest'),
prompt: 'Try the pro model first...',
use: [
fallback({
models: [googleAI.model('gemini-flash-latest')], // try flash if pro fails
statuses: ['RESOURCE_EXHAUSTED']
})
]
});
8. Context Compression Middleware (contextCompression)
Compresses conversation context when it grows too large, reducing token usage and costs in agentic workflows and tool loops. Compression triggers when estimated tokens or previous turn usage exceeds maxInputTokens.
Strategies applied:
- Safety cap: Hard-truncates any single oversized tool response (
maxToolResponseChars, default: 400,000 chars) on every turn with a[TRUNCATED: ...]marker. - Deduplication: Replaces duplicate tool responses with a short notice (
deduplicateToolResponses). - Tool response truncation: Truncates tool responses exceeding a character limit (
toolResponses), preserving the most recent tool response messages and appending a[Truncated N characters]marker. - Message count cap: Drops older non-system messages when exceeding
maxMessages, preserving system messages and ensuring conversation history begins with a user turn. - Summarization: Summarizes older conversation history using an LLM (
summarize), with optional threshold skipping (skipSummarizationThreshold) and dynamic overshoot adjustment. - Token budget fallback: If deduplication and tool response truncation cannot bring the context under
maxInputTokens, and no summary replaces older messages, older non-system messages are dropped down topreserveRecent. - Non-destructive session history: By default (
preserveOriginalMessages: true), compression state is saved in message metadata (message.metadata.contextCompression) rather than modifying original messages in place. Model calls receive the compressed view while session and UI logs retain full original messages. UseresolveCompressedHistory(messages)to extract active messages.
Ordering note: When combining
contextCompressionwith context-injecting middleware (such asfilesystem(),skills(), orartifacts()), placecontextCompressionafter those middlewares inuse: [...]so their injected instructions and tool outputs are accounted for during compression.
import { genkit } from 'genkit';
import { contextCompression, resolveCompressedHistory } from '@genkit-ai/middleware';
const ai = genkit({ ... });
const response = await ai.generate({
model: googleAI.model('gemini-flash-latest'),
prompt: 'Research and summarize...',
tools: [searchTool],
use: [
contextCompression({
maxInputTokens: 80000,
deduplicateToolResponses: { matchBy: 'name-and-input' },
toolResponses: { maxChars: 2000, preserveRecent: 2 },
summarize: {
model: googleAI.model('gemini-flash-lite-latest'),
preserveRecent: 6,
},
skipSummarizationThreshold: 0.25, // Skip LLM summary if cheap strategies save >= 25%
preserveOriginalMessages: true, // Non-destructive (default)
maxMessages: 20,
}),
],
});
// Original uncompressed history is preserved in response.messages:
console.log(response.messages.length);
// Resolve the active compressed messages sent to the model:
const activeMessages = resolveCompressedHistory(response.messages);
Options
| Option | Type | Default | Description |
|---|---|---|---|
maxInputTokens | number | Infinity | Triggers compression when token count exceeds this threshold. |
preserveRecent | number | 4 | Number of most recent non-system messages to preserve when dropping older messages (and default/fallback window for summarize.preserveRecent). |
preserveSystem | boolean | true | Always keep system instructions intact. |
maxToolResponseChars | number | 400000 | Hard cap on any single tool response size in characters (covers output and content together). Set <= 0 or Infinity to disable. |
deduplicateToolResponses | object | — | Deduplication settings for repeated tool calls. |
deduplicateToolResponses.matchBy | 'name-and-input' | 'name-only' | 'name-and-input' | How duplicate tool calls are matched. 'name-and-input' matches calls with identical tool name and arguments. 'name-only' groups all calls to a tool by name alone and discards earlier responses even when called with different inputs (use only for tools that return the latest state regardless of arguments). |
deduplicateToolResponses.keepRecent | number | 1 | Number of most recent responses to keep per duplicate group (minimum 1). |
deduplicateToolResponses.notice | string | standard text | Custom replacement text for deduplicated tool responses. |
toolResponses | object | — | Truncation settings for older tool responses. |
toolResponses.maxChars | number | — | Max characters per older tool response (covers output and content together, folding text content into output when truncated). |
toolResponses.preserveRecent | number | 2 | Number of most recent tool response messages to keep untruncated. |
summarize | object | — | LLM summarization settings (model, preserveRecent, prompt). |
summarize.model | ModelArgument | (required) | Model to use for summarization (model reference, model name string, or ModelAction). |
summarize.preserveRecent | number | 6 | Number of most recent non-system messages to keep un-summarized (defaults to top-level preserveRecent if set, and falls back to preserveRecent on shorter over-budget histories). |
summarize.prompt | string | standard prompt | Custom summarization prompt. Use {conversation} as a placeholder for the rendered messages to summarize. |
skipSummarizationThreshold | number | — | Skip summarization if cheap strategies save at least this fraction (0..1) of context and bring estimated tokens within maxInputTokens. |
maxMessages | number | — | Maximum message count target. Drops older non-system messages, ensuring history begins with a user turn. May be exceeded when system messages and the truncation notice fill the cap, since the newest turn is always kept. |
insertTruncationNotice | boolean | true | Inserts an advisory notice when messages are dropped. |
truncationNotice | string | standard text | Custom notice text to use when messages are dropped. |
preserveOriginalMessages | boolean | true | Preserves original uncompressed messages in request.messages and response.messages while storing compression state in message.metadata.contextCompression. Set to false to overwrite request.messages in place. |