Grafana Agent Observability TypeScript/JavaScript SDK
@grafana/agento11y records normalized LLM generation and tool-execution telemetry using your OpenTelemetry tracer/meter setup.
Installation
pnpm add @grafana/agento11y
For low-dependency runtimes that only need the core Agento11yClient and generation export APIs, use the slim core package:
pnpm add @grafana/agento11y-core
For a Grafana Cloud setup walkthrough (where to find the endpoint URL, instance ID, and API token), refer to the Grafana Cloud setup guide.
Validation
Run the shared core conformance suite for the JavaScript SDK from the repo root:
mise run test:ts:sdk-conformance
Run the cross-language aggregate core conformance suite from the repo root:
mise run sdk:conformance
Quick Start
The snippet below configures the SDK explicitly. As an alternative, set AGENTO11Y_* environment variables and call new Agento11yClient() with no arguments — refer to the Grafana Cloud setup guide for the variable names.
import { Agento11yClient } from "@grafana/agento11y";
const client = new Agento11yClient({
generationExport: {
protocol: "http",
endpoint: "https://agento11y-prod-<region>.grafana.net",
auth: {
mode: "basic",
tenantId: process.env.AGENTO11Y_AUTH_TENANT_ID,
basicPassword: process.env.AGENTO11Y_AUTH_TOKEN,
},
},
api: {
endpoint: "https://agento11y-prod-<region>.grafana.net",
},
});
await client.startGeneration(
{
conversationId: "conv-1",
model: { provider: "openai", name: "gpt-5" },
},
async (recorder) => {
const outputText = "Hello from model";
recorder.setResult({
output: [{ role: "assistant", content: outputText }],
});
}
);
await client.shutdown();
Content Capture
contentCapture controls what content the SDK includes in exported generation payloads and OTel span attributes. See Content Capture Modes for the canonical mode matrix and defaults; the snippets below show how to wire it up in TypeScript.
Client-level default:
import { Agento11yClient } from "@grafana/agento11y";
const client = new Agento11yClient({
contentCapture: "metadata_only",
});
The core SDK client treats "default" as "no_tool_content": generation content is captured but tool-execution arguments and results stay out of spans.
Per-generation override:
await client.startGeneration(
{
model: { provider: "openai", name: "gpt-5" },
contentCapture: "full",
},
async (recorder) => {
recorder.setResult({ output: [{ role: "assistant", content: "hi" }] });
}
);
Per-tool-execution override (here "full" opts into capturing tool arguments and results in the span):
await client.startToolExecution(
{ toolName: "search", contentCapture: "full" },
async (recorder) => {
recorder.setResult({ arguments: { q: "weather" }, result: { tempC: 18 } });
}
);
Dynamic resolution via contentCaptureResolver:
const client = new Agento11yClient({
contentCaptureResolver: (metadata) => {
if (metadata?.["tenant"] === "healthcare") {
return "metadata_only";
}
return "default"; // defer to `contentCapture`
},
});
The resolver receives the recording's metadata (or undefined for recording types that have no metadata, like tool executions). When a framework chain or retriever span ends with an error, it also calls the resolver with that run's metadata, to decide whether the error text is shown. Thrown errors are caught and treated as "metadata_only" (fail-closed).
Resolution precedence (highest to lowest):
- Per-recording
contentCaptureonGenerationStart/ToolExecutionStart contentCaptureResolverreturn value- Client-level
contentCapture(defaults to"no_tool_content")
Unlike the Go, Python, Java, and .NET SDKs, the JS SDK does not propagate the resolved capture mode through async context, so tool executions started inside a generation block do not automatically inherit the generation's mode. Set contentCapture on each ToolExecutionStart when you need a tool to follow a non-default policy.
User-provided metadata and tags are not stripped by any capture mode. SDK-internal metadata keys that carry content (e.g. call_error, agento11y.conversation.title) are stripped along with the matching content. See Tags and Metadata for where client tags, per-generation tags, metadata, and userId each show up (export vs spans vs metrics).
Pre-Ingest Redaction
Use generationSanitizer when you want to redact substrings from normalized generations before
validation, span sync, debug snapshots, and export.
Tool execution spans show the same conversation title, tool arguments, result, and error, so the
SDK runs the sanitizer over them too. It passes a generation with operationName: 'execute_tool'
that holds only that text: the title in conversationTitle, the arguments as a tool_call part,
and the result and the error as tool_result parts, the error one with isError: true. The
sanitizer is called once with the title when the span starts and once with the rest when it ends.
Text the sanitizer removes stays off the span. If a call throws, the text from that call stays
off the span; the span keeps the tool name and error category.
import {
Agento11yClient,
createSecretRedactionSanitizer,
} from "@grafana/agento11y";
const client = new Agento11yClient({
generationSanitizer: createSecretRedactionSanitizer({
redactInputMessages: false, // omit to fall back to AGENTO11Y_REDACT_INPUT_MESSAGES, then false
redactEmailAddresses: true,
}),
});
The built-in sanitizer:
- redacts high-confidence secret formats in assistant text and thinking
- redacts secret formats plus key/value secrets in system prompts, tool call inputs, and tool results
- redacts email addresses by default
- redacts the conversation title and call error
- redacts historic assistant turns and tool messages in input
- leaves user input unchanged unless input redaction is enabled
To preserve email addresses, opt out explicitly:
const client = new Agento11yClient({
generationSanitizer: createSecretRedactionSanitizer({
redactEmailAddresses: false,
}),
});
Configuring redaction via environment variables
createSecretRedactionSanitizer() reads AGENTO11Y_REDACT_INPUT_MESSAGES (accepts
1/0, true/false, yes/no, on/off) when redactInputMessages is omitted.
Precedence is explicit option > env var > false. An unrecognised env value is
warned and falls back to the next layer, so a typo cannot silently flip
redaction.
import {
createSecretRedactionSanitizer,
Agento11yClient,
} from "@grafana/agento11y";
// Omit redactInputMessages so AGENTO11Y_REDACT_INPUT_MESSAGES decides.
const client = new Agento11yClient({
generationSanitizer: createSecretRedactionSanitizer(),
});
Hooks and Guards
Use hooks when you want Agent Observability guard rules to run before an LLM call. The SDK evaluates the hook on your request path; guard rules configured in Grafana Cloud decide whether to allow, deny, or transform the input.
Hooks are disabled by default. Enable them on the client and call evaluateHook(...) before the provider request:
import { HookDeniedError, Agento11yClient } from "@grafana/agento11y";
const client = new Agento11yClient({
hooks: { enabled: true, phases: ["preflight"], timeoutMs: 15_000, failOpen: true },
});
let messages = [{ role: "user" as const, content: "Summarize this customer note..." }];
const response = await client.evaluateHook({
phase: "preflight",
context: {
agentName: "support-agent",
agentVersion: "1.0.0",
model: { provider: "openai", name: "gpt-5" },
conversationId: "support-case-42",
},
input: {
messages,
systemPrompt: "You are a helpful support agent.",
conversationPreview: "Summarize this customer note...",
},
});
if (response.action === "deny") {
throw new HookDeniedError(response.reason ?? "", response.ruleId, response.evaluations);
}
messages = response.transformedInput?.messages ?? messages;
With failOpen: true, hook transport errors resolve to allow so an unavailable evaluator does not block production traffic. Set failOpen: false for strict paths that should fail closed.
Set hooks.strictTransforms: true to reject malformed or empty transformed_input before the decoder discards data. The default is false, which preserves permissive decoding. Strict mode accepts the same valid SDK, snake_case, and Go/protobuf encodings. A supplied null or empty transform, empty collections, and unsupported transform fields throw HookTransformError. Omitting transformed_input remains valid.
Import HookTransformError from @grafana/agento11y or @grafana/agento11y-core. The error contains no response payload, source input, or authentication headers. The SDK throws the error even with failOpen: true; callers must not treat it as an evaluator outage or forward the original input. Strict decoding validates shape, not whether your integration can safely apply a rewrite. Refuse transforms your integration cannot apply.
Override strict mode for one evaluation with client.evaluateHook(request, { strictTransforms: true }). The override does not change the client configuration.
Set context.conversationId to the same ID used by startGeneration(...). The SDK also reads withConversationId(...) and the active OpenTelemetry span when explicit correlation fields are omitted. This lets Agent Observability retain denied preflight attempts even though no LLM generation is created.
If you use transformed input, pass the transformed messages/system prompt to the provider and record those same values in startGeneration(...). If you use the Vercel AI SDK adapter, see docs/frameworks/vercel-ai-sdk.md for automatic preflight hook wiring.
Configure OTEL exporters (traces/metrics) in your application OTEL SDK setup. You can optionally pass tracer and meter directly to Agento11yClient.
Quick OTEL setup pattern before creating the agento11y client:
import { NodeSDK } from "@opentelemetry/sdk-node";
const otel = new NodeSDK();
await otel.start();
See OpenTelemetry Setup for the full wiring, including the OTLP metric exporter, and why analytics stays empty when this step is skipped.
Core API
startGeneration(...)andstartStreamingGeneration(...)client.startToolExecution(...)andGenerationRecorder.startToolExecution(...)- Recorder methods:
setResult(...),setCallError(...),end(),getError() - Lifecycle:
flush(),shutdown()
Manual try/finally style
const recorder = client.startGeneration({
model: { provider: "anthropic", name: "claude-sonnet-4-5" },
});
try {
recorder.setResult({
output: [{ role: "assistant", content: "Done" }],
});
} catch (error) {
recorder.setCallError(error);
throw error;
} finally {
recorder.end();
}
Embedding Observability
Use startEmbedding(...) for embedding API calls. Embedding recording creates OTel spans and SDK metrics only, and does not enqueue generation exports.
await client.startEmbedding(
{
agentName: "retrieval-worker",
agentVersion: "1.0.0",
model: { provider: "openai", name: "text-embedding-3-small" },
},
async (recorder) => {
const response = await openai.embeddings.create(request);
recorder.setResult({
inputCount: request.input.length,
inputTokens: response.usage?.prompt_tokens ?? 0,
inputTexts: request.input,
responseModel: response.model,
});
}
);
Input text capture is opt-in:
const client = new Agento11yClient({
embeddingCapture: {
captureInput: true,
maxInputItems: 20,
maxTextLength: 1024,
},
});
embeddingCapture.captureInput may expose PII/document content in spans. Keep it disabled by default and enable it only for scoped debugging.
TraceQL examples:
traces{gen_ai.operation.name="embeddings"}traces{gen_ai.operation.name="embeddings" && gen_ai.request.model="text-embedding-3-small"}traces{gen_ai.operation.name="embeddings" && error.type!=""}
Tool Execution Example
Use GenerationRecorder.startToolExecution(...) to parent a tool span to that generation, including across await, without a global context manager. The method does not change active context or inherit generation capture settings, identity, or request model. Set those tool options explicitly when they must match the generation. client.startToolExecution(...) remains available and uses the active OpenTelemetry context.
await client.startGeneration(
{
conversationId: "weather-session",
agentName: "weather-agent",
model: { provider: "openai", name: "gpt-5" },
tools: [{ name: "weather", type: "function" }],
contentCapture: "metadata_only",
},
async (generation) => {
const city = await Promise.resolve("Paris");
const tool = generation.startToolExecution({
toolName: "weather",
conversationId: "weather-session",
agentName: "weather-agent",
requestProvider: "openai",
requestModel: "gpt-5",
contentCapture: "metadata_only",
});
try {
tool.setResult({ arguments: { city }, result: { tempC: 18 } });
} catch (error) {
tool.setCallError(error);
throw error;
} finally {
tool.end();
}
generation.setResult({ output: [{ role: "assistant", content: "18°C in Paris" }] });
}
);
The generation callback ends the generation automatically. The tool factory returns a recorder; call its end() yourself. Both recordings above omit content under metadata_only. Configure OpenTelemetry providers to retain the spans; explicit parentage does not create exporters.
Tracing skill loads, resource reads, and commands
Set ToolExecutionStart.executionKind explicitly to "load_skill", "read_skill_resource", or "command".
The ToolExecutionKind type defines these values. The SDK does not infer execution kinds from tool names, paths, or arguments.
Skill and process attributes follow merged GenAI #498, which remains in development.
| Execution kind | Meaning | Applicable fields |
|---|---|---|
"load_skill" | Load skill instructions | skillName, skillDescription, skillSourceURI |
"read_skill_resource" | Read a skill resource | Skill fields above, skillResourceName |
"command" | Execute a command, optionally for a skill | Skill/resource fields above, processExecutableName, processExecutablePath; completion processExitCode |
Unset or unrecognized execution kinds keep generic tool behavior and emit no skill or process attributes. The SDK ignores inapplicable fields.
Skill fields emit gen_ai.skill.name, gen_ai.skill.description, gen_ai.skill.source.uri, and gen_ai.skill.resource.name.
Process fields emit process.executable.name, process.executable.path, and process.exit.code.
The SDK does not dual-write agento11y.skill.name.
Span names start with execute_tool <tool> and use trimmed, nonblank identity:
load_skillappends the skill name, when present.read_skill_resourceappends<skill> <resource>, or just<resource>without a skill. Without a resource, it appends nothing.commanduses the same resource suffix, then falls back to the executable name, then no suffix.
Skill resource reads and commands never use a skill-only suffix. The span name and applicable structural names are available to samplers at creation. Private fields and the completion exit code are excluded from sampler attributes.
Process fields describe one directly executed process. Do not infer them from shell text, pipelines, or command batches.
ToolExecutionResult.processExitCode is an optional integer-valued number: omission means unknown; zero records successful completion.
Only safe integers are emitted. Booleans, strings, and fractional values are not coerced into exit codes.
Exit codes are independent of recorded execution errors.
For example, record a known completion code as follows:
await client.startToolExecution(
{
toolName: "exec",
executionKind: "command",
skillName: "review",
skillResourceName: "scripts/check.py",
processExecutableName: "python3",
},
async (recorder) => {
recorder.setResult({ processExitCode: 0 });
}
);
skillDescription, skillSourceURI, and processExecutablePath are private content.
The SDK omits them from spans under effective "metadata_only" or "full_with_metadata_spans", after resolving overrides and the resolver.
Core "no_tool_content" retains these private fields; arguments/results still require the existing content opt-in.
Every exported skill source URI has userinfo, query, and fragment removed, even under "full".
Invalid or relative URIs are omitted without network or filesystem lookup.
When debug history is enabled, capture modes do not sanitize tool snapshots: raw arguments, results, and error text remain there.
Generic tool recording and executeToolCalls remain unchanged.
For load_skill, read_skill_resource, and command, gen_ai.client.operation.duration adds agento11y.tool.execution.kind.
It also includes a trimmed, nonblank gen_ai.skill.name when supplied, including failed attempts and reduced capture modes.
No resource or process metric dimensions are added.
See the client metric contract.
Provider Helpers
- OpenAI:
docs/providers/openai.md - Anthropic:
docs/providers/anthropic.md - Gemini:
docs/providers/gemini.md
Framework Handlers
Use module subpath exports for framework callback integrations:
- LangChain:
@grafana/agento11y/langchain - LangGraph:
@grafana/agento11y/langgraph - OpenAI Agents:
@grafana/agento11y/openai-agents - LlamaIndex:
@grafana/agento11y/llamaindex - Google ADK:
@grafana/agento11y/google-adk - Vercel AI SDK:
@grafana/agento11y/vercel-ai-sdk - Strands Agents:
@grafana/agento11y/strands - LangChain guide:
docs/frameworks/langchain.md - LangGraph guide:
docs/frameworks/langgraph.md - OpenAI Agents guide:
docs/frameworks/openai-agents.md - LlamaIndex guide:
docs/frameworks/llamaindex.md - Google ADK guide:
docs/frameworks/google-adk.md - Vercel AI SDK guide:
docs/frameworks/vercel-ai-sdk.md - Strands Agents guide:
docs/frameworks/strands.md
import { Agento11yClient } from "@grafana/agento11y";
import { withAgento11yLangChainCallbacks } from "@grafana/agento11y/langchain";
import { withAgento11yLangGraphCallbacks } from "@grafana/agento11y/langgraph";
import { withAgento11yOpenAIAgentsHooks } from "@grafana/agento11y/openai-agents";
import { withAgento11yLlamaIndexCallbacks } from "@grafana/agento11y/llamaindex";
import { withAgento11yGoogleAdkPlugins } from "@grafana/agento11y/google-adk";
import { createAgento11yVercelAiSdk } from "@grafana/agento11y/vercel-ai-sdk";
import { withAgento11yStrandsHooks } from "@grafana/agento11y/strands";
import { Runner } from "@openai/agents";
import { CallbackManager } from "llamaindex";
const client = new Agento11yClient();
const langChainConfig = withAgento11yLangChainCallbacks(undefined, client, { providerResolver: "auto" });
const langGraphConfig = withAgento11yLangGraphCallbacks(undefined, client, { providerResolver: "auto" });
const runner = new Runner();
const openAIAgentsHooks = withAgento11yOpenAIAgentsHooks(runner, client, { providerResolver: "auto" });
const callbackManager = new CallbackManager();
const llamaIndexConfig = withAgento11yLlamaIndexCallbacks({ callbackManager }, client, { providerResolver: "auto" });
const googleAdkRunnerConfig = withAgento11yGoogleAdkPlugins(undefined, client, { providerResolver: "auto" });
const vercelAiSdk = createAgento11yVercelAiSdk(client, { agentName: "vercel-agent" });
const strandsConfig = withAgento11yStrandsHooks(undefined, client, { conversationId: "chat-123" });
Framework handlers use the Agento11yClient instance you pass in. If that client is configured with
generationSanitizer, the same redaction policy applies automatically to generations recorded
through LangChain, LangGraph, OpenAI Agents, LlamaIndex, Google ADK, and Vercel AI SDK integrations.
The same redaction policy also applies to Strands Agents generations.
Each framework handler injects:
agento11y.framework.name(langchain,langgraph,openai-agents,llamaindex,google-adk,vercel-ai-sdk, orstrands)agento11y.framework.source(handlerfor existing callback handlers,frameworkfor Vercel AI SDK hooks,hooksfor Strands)agento11y.framework.language(javascriptfor existing callback handlers,typescriptfor Vercel AI SDK and Strands hooks)metadata["agento11y.framework.run_id"]metadata["agento11y.framework.thread_id"](when present)metadata["agento11y.framework.parent_run_id"](when available)metadata["agento11y.framework.component_name"]metadata["agento11y.framework.run_type"]metadata["agento11y.framework.tags"]metadata["agento11y.framework.retry_attempt"](when available)metadata["agento11y.framework.event_id"](when available)metadata["agento11y.framework.langgraph.node"](LangGraph when available)
Conversation mapping is conversation-first:
conversation_id/session_id/group_idfrom framework context first- then
thread_id - deterministic fallback
agento11y:framework:<framework_name>:<run_id>
When present in generation metadata, low-cardinality framework keys are copied onto generation span attributes.
For LangGraph persistence, pass configurable.thread_id and reuse it across invocations:
const threadConfig = {
...withAgento11yLangGraphCallbacks(undefined, client, { providerResolver: "auto" }),
configurable: { thread_id: 'customer-42' },
};
await graph.invoke({ prompt: 'Remember my timezone is UTC+1.', answer: '' }, threadConfig);
await graph.invoke({ prompt: 'What timezone did I give you?', answer: '' }, threadConfig);
Behavior
- Generation modes are explicit:
SYNCandSTREAM. - Generation export supports HTTP, gRPC, and
none(instrumentation-only). - Traces/metrics use
config.tracer/config.meterwhen provided, otherwise OTEL globals. - Exports are asynchronous with bounded queueing and retry/backoff. Each export request defaults to a 30-second timeout.
flush()drains queued generations.shutdown()runs a final flush with a deadline equal to the configured request timeout. When that deadline passes, it discards the queued exports without a retry and closes the exporter.- Empty tool names produce a no-op tool recorder.
- Generation/tool spans always include SDK identity attributes:
agento11y.sdk.name=sdk-js
- Normalized generation metadata always includes the same SDK identity key; conflicting caller values are overwritten.
- Raw provider artifacts are opt-in (
rawArtifacts: true).
Instrumentation-only mode (no generation send)
Set generationExport.protocol to "none" to keep generation/tool instrumentation and spans while disabling generation transport.
const client = new Agento11yClient({
generationExport: {
protocol: "none",
},
});
SDK metrics
The SDK emits these OTel histograms through your configured OTEL meter provider:
gen_ai.client.operation.durationgen_ai.client.token.usagegen_ai.client.time_to_first_tokengen_ai.client.tool_calls_per_operation
Generation export auth modes
Auth is configured for generationExport.
mode: "none"mode: "tenant"(requirestenantId, injectsX-Scope-OrgID)mode: "bearer"(requiresbearerToken, injectsAuthorization: Bearer <token>)mode: "basic"(requiresbasicPassword+basicUserortenantId, injectsAuthorization: Basic <base64(user:password)>; also injectsX-Scope-OrgIDwhentenantIdis set — for multi-tenant deployments only, not needed for Grafana Cloud)
Invalid mode/field combinations throw during client config resolution.
If explicit headers already contain Authorization or X-Scope-OrgID, explicit headers take precedence.
const client = new Agento11yClient({
generationExport: {
protocol: "http",
endpoint: "https://agento11y-prod-<region>.grafana.net",
auth: {
mode: "basic",
tenantId: process.env.AGENTO11Y_AUTH_TENANT_ID,
basicPassword: process.env.AGENTO11Y_AUTH_TOKEN,
},
},
api: {
endpoint: "https://agento11y-prod-<region>.grafana.net",
},
});
Grafana Cloud auth (basic)
For Grafana Cloud, use basic auth mode. The username is your Grafana Cloud instance/tenant ID and the password is your Grafana Cloud API key. See the Grafana Cloud Agent Observability getting started docs for full setup steps; for this SDK endpoint, copy the API URL from Observability → Agent Observability → Configuration. It looks like https://agento11y-prod-<region>.grafana.net.
const client = new Agento11yClient({
generationExport: {
protocol: "http",
endpoint: "https://agento11y-prod-<region>.grafana.net",
auth: {
mode: "basic",
tenantId: process.env.AGENTO11Y_AUTH_TENANT_ID,
basicPassword: process.env.AGENTO11Y_AUTH_TOKEN,
},
},
});
If your deployment requires a distinct username, set basicUser explicitly:
auth: {
mode: "basic",
tenantId: process.env.AGENTO11Y_AUTH_TENANT_ID,
basicUser: process.env.AGENTO11Y_AUTH_TENANT_ID,
basicPassword: process.env.AGENTO11Y_AUTH_TOKEN,
},
Export timeout
AGENTO11Y_EXPORT_TIMEOUT_MS sets the timeout for each generation or workflow-step export request. It accepts base-10 integers from 1 through 2147483647 and defaults to 30000 milliseconds. An explicit generationExport.timeoutMs value wins over the environment variable.
shutdown() uses the same resolved timeout as the deadline for its final flush, so a request that is already in flight gets one complete attempt.
Export retry and queue tuning
Generation and workflow-step exports retry five times by default, with exponential backoff capped at five seconds. Set AGENTO11Y_MAX_RETRIES and AGENTO11Y_MAX_BACKOFF_MS to base-10 integers from 1 through 2147483647 to override those defaults. Explicit generationExport.maxRetries and generationExport.maxBackoffMs values win over the environment variables.
Each in-memory export queue holds 2,000 records by default. Set AGENTO11Y_QUEUE_SIZE to a base-10 integer from 1 through 2147483647 to override that capacity. Size it from the peak records per second multiplied by the expected outage in seconds, plus headroom. Generations and workflow steps have separate queues of this capacity. Larger queues consume application memory and do not survive a process restart. An explicit generationExport.queueSize value wins over the environment variable.
Retries run in the background exporter. While an export is retrying, the in-memory queue can fill and newer generations can be dropped. Use a retry budget sized for a short, known interruption rather than a permanent multi-minute setting.
Wiring custom env vars
The SDK only auto-loads AGENTO11Y_* env vars (AGENTO11Y_ENDPOINT, AGENTO11Y_PROTOCOL, AGENTO11Y_AUTH_MODE, AGENTO11Y_AUTH_TOKEN, etc.) when you call new Agento11yClient(). For any other env var (for example one your secret manager exposes under a different name), read it in your app and pass the value into the config:
const generationBearerToken = (process.env.MY_APP_AGENTO11Y_TOKEN ?? "").trim();
const client = new Agento11yClient({
generationExport: {
protocol: "http",
endpoint: "https://agento11y-prod-<region>.grafana.net",
auth:
generationBearerToken.length > 0
? { mode: "bearer", bearerToken: generationBearerToken }
: {
mode: "basic",
tenantId: process.env.AGENTO11Y_AUTH_TENANT_ID,
basicPassword: process.env.AGENTO11Y_AUTH_TOKEN,
},
},
api: {
endpoint: "https://agento11y-prod-<region>.grafana.net",
},
});
Common topology:
- Grafana Cloud: generation
basicmode with instance ID and API key. - Self-hosted direct to the ingest API: generation
tenantmode. - Traces/metrics via OTEL Collector/Alloy: configure exporters in your app OTEL SDK setup.
- Enterprise proxy: generation
bearermode to proxy; proxy authenticates and forwards tenant header upstream.
Conversation Ratings
Use the SDK helper to submit user-facing ratings:
const result = await client.submitConversationRating("conv-123", {
ratingId: "rat-123",
rating: "CONVERSATION_RATING_VALUE_BAD",
comment: "Answer ignored user context",
metadata: { channel: "assistant-ui" },
source: "sdk-js",
});
console.log(result.rating.rating, result.summary.hasBadRating);
submitConversationRating sends requests to api.endpoint, which should be the Grafana Cloud Agent Observability API URL from Agent Observability configuration, and uses the same generation-export auth headers already configured on the SDK client.
Experiments
Offline evals: run an agent over a dataset, grade it, and publish runs, trials, and scores to Agent Observability. The API lives on a separate subpath, so importing the core package never pulls experiments code into an edge-runtime bundle:
import { ExperimentsClient, withExperiment } from "@grafana/agento11y/experiments";
const client = new ExperimentsClient({
endpoint: process.env.AGENTO11Y_ENDPOINT,
tenantId: process.env.AGENTO11Y_AUTH_TENANT_ID,
ingestToken: process.env.AGENTO11Y_AUTH_TOKEN,
});
const suite = {
suiteId: "smoke",
name: "Smoke",
testCases: [{ testCaseId: "add", input: "2+2", expected: "4" }],
};
const verifier = { evaluatorId: "exact", version: "1", kind: "deterministic" };
await withExperiment(client, { experimentId: "run-1", name: "smoke run", suite }, async (experiment) => {
for (const testCase of suite.testCases) {
await experiment.withTrial(testCase, async (trial) => {
const answer = await runAgent(testCase.input);
trial.recordIO({ input: testCase.input, output: answer });
trial.finalScore(answer === testCase.expected, { evaluator: verifier });
});
}
});
withExperiment finalizes the run on both the success and the failure path, and
withTrial closes the trial and terminalizes it as errored when the callback
throws. Use Experiment.start, trial.start(), and close() directly when a
runner owns its own control flow.
An evaluator stored in your tenant can also grade a trial, instead of a score
computed in the runner. That path is experimental and requires
AGENTO11Y_ENABLE_EXPERIMENTAL_FEATURES=true:
await experiment.withTrial(testCase, async (trial) => {
const { conversationId } = await runInstrumentedAgent(testCase.input);
trial.bindConversation(conversationId);
await trial.evaluate("helpfulness");
});
See docs/experiments.md for the full surface: score
kinds, local judges, artifacts, reports, portable and stored suites, cross-process
trials, and experimental OpenTelemetry trial telemetry.
Promptfoo users can publish a completed eval through its afterAll hook with
createPromptfooExtension from @grafana/agento11y/promptfoo. See the
Promptfoo integration guide and the
runnable example.