dt-evals - Dynatrace Evaluation CLI tool

Open source continuous evals for LLM applications, with production prompt traces as the dataset.

📺 See it in action: How to Run LLM Evaluations with dt-evals — a hands-on walkthrough of pulling production traces, scoring them with an LLM judge, and reading the results back in Dynatrace.

dt-evals is for teams shipping chat, RAG, copilots, and agent workflows who want evals to run on real traffic, not just curated test sets.

Today, the CLI uses Dynatrace as both the trace source and the evaluation result store. It pulls gen_ai.* spans from your live environment, masks sensitive data in memory, scores real production interactions with an LLM judge, detects score drift over time, and writes structured evaluation results back to Dynatrace as business events.

That means when a score drops or an outlier appears, you do not just see that quality regressed, you can trace it back through the full AI execution path: prompts, retrieval context, model calls, tool usage, latency, failures, and service dependencies.

In practice, this gives AI engineers a closed loop for evaluation, observability, alerting, anomaly detection, and remediation on the same production telemetry they already use to operate their systems.

The repository also includes dt-eval-lib, a reusable TypeScript evaluation engine for running the same judge-based metrics directly in code, using either the built-in evaluator catalog or your own custom prompt definitions. It also fits cleanly into broader evaluation workflows and observability stacks such as Ragas, MLflow, or Langfuse when you want to orchestrate or track evals in those systems alongside this library.

npx @dynatrace-oss/dt-evals configure
npx @dynatrace-oss/dt-evals validate
npx @dynatrace-oss/dt-evals run --since 1h --sample 10 --concurrency 10

Tune --concurrency (or judge.concurrency in your yaml) to control how many judge calls run in parallel — bump it for faster runs, drop it if the provider rate-limits you. Default is 5.

🎮 See it live before installing — open the dt-evals playground dashboard on our public Dynatrace tenant. Real evaluation runs against production GenAI traces — scores per metric, drift over time, threshold breaches, click-through to the originating trace.

Why Teams Use It

• Run evals on real production traces, not static test sets
• Keep eval results in Dynatrace with traces, logs, metrics, dashboards, and alerts
• Catch common LLM and agent failure modes with built-in judge metrics
• Detect regressions, drift, and outlier score changes early
• Go from low score to root cause with end-to-end AI observability
• Correlate failures with prompts, retrieval, tool calls, latency, and dependencies
• Reuse the same evaluator catalog in CI, app code, and local workflows with dt-eval-lib
• Trigger alerts and optional remediation from the same evaluation pipeline

Packages

What It Does

• 15 built-in judge metrics for safety, grounding, relevance, quality, and retrieval quality
• Statistical drift detection against a 7 day baseline of prior evaluation scores
• Flexible sampling with random percentage, latest N, or errors-only
• PII masking before judge calls for emails, phone numbers, credit cards, and SSNs
• Evaluated prompt/response excluded from bizevents by default — opt in with storeEvaluatedPrompt (see Configuration)
• OpenAI, Anthropic, Azure OpenAI, Google Gemini Enterprise Platform (Vertex AI), and Bedrock support with optional custom base URLs for gateways and proxies
• CI friendly runs with JSON output and non-zero exit codes on threshold breaches
• Local run history with list, inspect, and export flows
• Scheduled runs stored locally and triggerable on demand
• Evaluator inspection and testing from the CLI
• TypeScript API for running the evaluator catalog directly in code

How It Works

Dynatrace spans (gen_ai.*)
        |
        v
sample traces -> mask sensitive data -> score with LLM judge -> write business events
                                                                    |
                                                                    v
                                             query in DQL, dashboard, alert, export

The current CLI path is Dynatrace specific. The product positioning is broader: continuous evals for AI-native engineering teams, using your live LLM traffic as the source of truth.

Requirements

• Node.js >=20
• A Dynatrace environment with GenAI spans or OpenTelemetry-style gen_ai.* fields
• Credentials for your judge provider (OpenAI, Anthropic, Azure OpenAI, Google Gemini Enterprise Platform (Vertex AI), or Bedrock)

Install

npm install -g @dynatrace-oss/dt-evals

Or run without installing:

npx @dynatrace-oss/dt-evals <command>

The CLI also auto-loads a local .env file from the current working directory.

Quick Start

1. Create config

Interactive setup:

dt-evals configure

Non-interactive setup:

dt-evals configure \
  --env-url https://your-env.live.dynatrace.com \
  --api-token "$DT_API_TOKEN" \
  --provider openai \
  --api-key "$OPENAI_API_KEY" \
  --model gpt-4.1 \
  --since 1h \
  --output .dt-eval.yaml

2. Validate the setup

dt-evals validate

This checks:

• config schema
• Dynatrace connectivity
• judge provider reachability
• evaluator catalog availability

3. Run evals on recent traces

dt-evals run --since 1h --sample 10 --concurrency 5

Run only one evaluator:

dt-evals run --since 6h --metric faithfulness

Preview the work without calling the judge or writing results:

dt-evals run --since 1h --sample 5 --dry-run

CLI Workflows

Production eval run

dt-evals run \
  --since 2h \
  --sample 20 \
  --concurrency 8 \
  --debug

CI gate on quality regressions

dt-evals run --since 6h --metric relevance --ci

• exit code 0: no threshold breaches
• exit code 1: one or more threshold breaches

Example GitHub Actions step:

- name: Run LLM eval gate
  run: npx @dynatrace-oss/dt-evals run --since 6h --metric faithfulness --ci
  env:
    DT_ENV_URL: ${{ secrets.DT_ENV_URL }}
    DT_API_TOKEN: ${{ secrets.DT_API_TOKEN }}
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Run drift detection only

dt-evals run --metric drift --since 24h

This uses prior evaluation results already written to Dynatrace and compares the recent window against a 7 day baseline.

Inspect available evaluators

dt-evals evaluators list
dt-evals evaluators show faithfulness
dt-evals evaluators test relevance

Create or remove a custom evaluator:

dt-evals evaluators add
dt-evals evaluators delete my-custom-eval

Manage run history

dt-evals runs list --limit 20
dt-evals runs show run-2026-04-10T12-00-00-ab12cd34
dt-evals runs export --format csv --output runs.csv

Run records are stored locally in ~/.dt-eval/runs.json.

Schedule recurring runs

dt-evals schedule add --name hourly-rag --cron "0 * * * *" --since 1h --sample 10
dt-evals schedule list
dt-evals schedule run <schedule-id>
dt-evals schedule disable <schedule-id>
dt-evals schedule enable <schedule-id>
dt-evals schedule delete <schedule-id>

Schedules are stored locally in ~/.dt-eval/schedules.json.

Check current status

dt-evals status
dt-evals configure --show

Commands

Global flags:

--verbose    Enable verbose logs
--json       Emit structured JSON logs

run flags:

--config <path>        Path to config file
--since <duration>     Trace lookback window, for example 1h or 24h
--sample <percent>     Percentage of traces to evaluate
--metric <name>        Run only one evaluator
--dry-run              Do not call the judge or write results
--ci                   JSON result output and exit 1 on threshold breach
--concurrency <n>      Parallel judge calls (overrides judge.concurrency in the config; default 5)
--store-evaluated-prompt   Include the evaluated prompt/response in bizevents (overrides storeEvaluatedPrompt in the config; default: false)
--debug                Per-step timing logs

Configuration

Config is resolved in this order:

Example config

schemaVersion: 3
name: travel-assistant-prod

dynatrace:
  environmentUrl: https://your-env.live.dynatrace.com
  dtctlContext: my-prod-context

judge:
  provider: openai
  model: gpt-4.1
  timeout: 30000
  maxRetries: 2
  concurrency: 5

scope:
  service: travel-assistant
  since: 1h
  operationNames: [chat, text_completion, generate_content]
  sampling:
    strategy: random
    percent: 10

metrics:
  enabled:
    - faithfulness
    - hallucination
    - relevance
    - answer-completeness
    - id: valid-json
      method: json_schema
      params:
        schema:
          type: object
          required: [answer]
    - id: no-refusal
      method: must_not_contain
      params:
        keywords: ["i cannot help", "i'm sorry"]
        mode: any
    - drift

alerts:
  thresholds:
    faithfulness: 0.7
    relevance: 0.7
    answer-completeness: 0.8

storeEvaluatedPrompt: false

String entries and object entries without method use the configured LLM-as-judge provider. Deterministic entries set method to exact_match, regex, must_not_match, json_schema, must_contain, or must_not_contain. exact_match compares the span output against a literal params.expectedOutput defined in the config:

metrics:
  enabled:
    - id: exact-answer
      method: exact_match
      params:
        expectedOutput: "OK"
        caseSensitive: false
        trim: true

By default, the runner keeps only chat/text-generation spans: chat, text_completion, and generate_content. Configure scope.operationNames to narrow or extend that keep-list, for example operationNames: [chat]. Set it to [] only if you intentionally want to disable the operation-name filter.

Sampling strategies:

Cross-tenant configuration

dynatrace.environmentUrl / dynatrace.apiToken describe a single tenant that handles both reads and writes. To split them — fetch GenAI spans from one tenant and write evaluation bizevents to another — use origin (read) and destination (write):

dynatrace:
  origin:
    environmentUrl: https://prod.live.dynatrace.com
  destination:
    environmentUrl: https://eval-results.dev.apps.dynatracelabs.com

Top-level environmentUrl / apiToken (if present) act as fallbacks — if either side omits a field, the top-level value is used. So a single-tenant config is just the cross-tenant form with both sides empty.

The validate command probes each side separately, including a real fetch spans | limit 1 against the origin to catch missing storage:spans:read scope before a run is attempted.

Required token scopes

For dt-evals alerts apply (Dynatrace Workflows), grant these additionally on the dtctl OAuth client:

Custom span field mapping

By default, the CLI reads OTel GenAI semconv (gen_ai.input.messages, gen_ai.output.messages, …). It also probes a small set of legacy fallback attributes (gen_ai.prompt.N.*, gen_ai.completion.0.content) for compatibility with older emitters. If your spans expose the LLM I/O under different attribute names — for example OpenInference (llm.input_messages, llm.output_messages, llm.model_name) — point the CLI at them via scope.spanFields. See the dynatrace-ai-agent-instrumentation-examples for end-to-end instrumentation examples. Each entry accepts a single attribute or a list of candidates; the first non-null value wins, with the built-in defaults appended as fallback.

Example 1 — explicit legacy output mapping:

scope:
  service: pydantic-ai-music-agent
  spanFields:
    output: gen_ai.completion.0.content

The default output mapping uses gen_ai.output.messages. spanFields lets you opt into legacy or emitter-specific attributes explicitly.

Example 2 — OTel spans not following the GenAI SemConv (OpenInference):

scope:
  service: my-openinference-service
  since: 30m
  spanFields:
    input: [llm.input_messages, input.value]
    output: [llm.output_messages, output.value]
    model: llm.model_name

OpenInference usually stores chat turns in llm.input_messages / llm.output_messages and plain-text payloads in input.value / output.value, so listing both keeps the mapping resilient across instrumentations.

Filtering spans by attribute

scope.filters narrows the DQL fetch to spans carrying specific attribute values. Keys are span attribute names; each value is a single value or a list. Separate keys are ANDed together, and a list of values for one key is ORed via in(). Filtering happens in Grail, so excluded spans are never transferred or judged.

Example 1 — isolate the orchestrator in a multi-agent service:

scope:
  service: support-orchestrator
  filters:
    gen_ai.agent.name: router

service alone cannot separate an orchestrator from the sub-agents it invokes when both run in the same service and emit the same gen_ai.operation.name.

Example 2 — restrict to tagged production traffic:

scope:
  service: travel-assistant
  filters:
    deployment.environment: production
    gen_ai.request.model: [gpt-4.1, gpt-4.1-mini]

Attribute names are passed to DQL verbatim, so an unknown or malformed attribute surfaces as a DQL error from Grail rather than being validated locally.

Per-metric input routing

Metric entries in metrics.enabled accept either a string id (the legacy form) or an object with inputs that overrides which canonical span field feeds each evaluator input slot. This is useful when a metric should score only part of the conversation — e.g. user-frustration evaluates the user's turn in isolation, not the joined transcript:

metrics:
  enabled:
    - faithfulness                                  # plain string form
    - id: user-frustration
      inputs:
        input: userPrompt
    - id: hallucination
      inputs:
        context: context

Available canonical fields: input, output, context, systemInstruction, model, userPrompt (latest user-role prompt slot, extracted from gen_ai.prompt.N.role == "user"). context has no built-in default span attribute — if a span does not provide it, evaluator context is omitted unless you explicitly route inputs.context elsewhere.

Full example

A complete .dt-eval.yaml combining everything — schema bump, custom span field mapping, per-metric input routing, sampling, alerts:

schemaVersion: 3
name: travel-assistant-prod

dynatrace:
  environmentUrl: https://your-env.live.dynatrace.com
  # apiToken loaded from DT_API_TOKEN env var

judge:
  provider: azure-openai
  model: gpt-4.1-mini
  # apiKey, baseUrl, apiVersion loaded from AZURE_OPENAI_* env vars

scope:
  service: travel-assistant
  since: 1h
  operationNames: [chat, text_completion, generate_content]
  sampling:
    strategy: latest
    count: 50
  spanFields:
    context: span.context
    # input, systemInstruction, model use built-in defaults

metrics:
  enabled:
    - faithfulness
    - relevance
    - hallucination
    - answer-completeness
    - id: user-frustration
      inputs:
        input: userPrompt
    - id: no-refusal
      method: must_not_contain
      params:
        keywords: ["i cannot help", "i'm sorry"]
        mode: any

alerts:
  thresholds:
    faithfulness: 0.7
    relevance: 0.7
    answer-completeness: 0.8
    user-frustration: 1

Useful environment variables:

DT_ENV_URL=https://your-env.live.dynatrace.com
DT_API_TOKEN=dt0c01.xxxxx
DT_DTCTL_CONTEXT=my-prod-context

# Cross-tenant overrides (optional — when set, these win over the top-level pair)
DT_ORIGIN_ENV_URL=https://prod.live.dynatrace.com
DT_ORIGIN_API_TOKEN=dt0s16.xxxxx
DT_DESTINATION_ENV_URL=https://eval-results.dev.apps.dynatracelabs.com
DT_DESTINATION_API_TOKEN=dt0s16.yyyyy

JUDGE_PROVIDER=openai
JUDGE_MODEL=gpt-4.1

OPENAI_API_KEY=sk-...
OPENAI_BASE_URL=https://your-gateway.example.com/v1

ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_BASE_URL=https://your-proxy.example.com

# Google Gemini Enterprise Platform (Vertex AI)
GOOGLE_API_KEY=...
# Or authenticate via Workload Identity / Application Default Credentials instead of an API key.
# Set the target project and region (or judge.project / judge.location in config):
GOOGLE_CLOUD_PROJECT=my-gcp-project
GOOGLE_CLOUD_LOCATION=global

Results in Dynatrace

Evaluation results are written as business events with event.type == "gen_ai.evaluation.result".

Important fields include:

Example DQL:

fetch bizevents
| filter event.type == "gen_ai.evaluation.result"
| summarize avg_score = avg(gen_ai.evaluation.score.value), by: { gen_ai.evaluation.name }
| sort avg_score asc

Drift results are written back as the same event type with gen_ai.evaluation.type == "drift".

Built-in Evaluators

dt-evals ships with 15 built-in LLM judge evaluators, plus drift detection as a separate statistical metric.

Deterministic evaluators (code evals)

Not every check needs an LLM. Add a method to a metric entry in metrics.enabled to run a fast, reproducible, zero-cost code eval (a deterministic check) instead of an LLM judge. Each one scores the span output as pass (1.0) or fail (0.0). Omitting method keeps the default LLM-as-judge behavior, so existing configs are unaffected.

Available methods: exact_match, regex, must_not_match, must_contain, must_not_contain, and json_schema.

exact_match — require the output to equal an expected string

metrics:
  enabled:
    - id: matches-canonical-answer
      method: exact_match
      params:
        expectedOutput: "APPROVED"
        trim: true            # strip surrounding whitespace before comparing
        caseSensitive: false  # default true; set false to ignore case

must_contain — require an expected keyword to be present

    - id: cites-a-source
      method: must_contain
      params:
        keywords: ["source", "reference", "according to"]
        mode: any          # any = pass if one is present; all = require every keyword

must_not_contain — block forbidden or harmful words

    - id: no-harmful-words
      method: must_not_contain
      params:
        keywords: ["harmful_word_1", "harmful_word_2", "harmful_word_3"]
        mode: any          # fail if ANY blocked word appears
        # matching is case-insensitive by default; set caseSensitive: true to change

must_not_match — fail if the output matches a pattern (e.g. a leaked IP address)

    - id: no-ip-leaked
      method: must_not_match
      params:
        pattern: "\\b(?:\\d{1,3}\\.){3}\\d{1,3}\\b"

regex — the opposite of must_not_match: pass only when the output matches

    - id: starts-with-ticket-id
      method: regex
      params:
        pattern: "^[A-Z]+-\\d+:"   # e.g. "AI-465: summary…"
        flags: "m"                 # ^ anchors each line, not just the string

json_schema — require valid, well-structured JSON

    - id: valid-response-object
      method: json_schema
      params:
        schema:
          type: object
          required: [answer]

Note: json_schema parses the raw output, so the output must be bare JSON — a response wrapped in markdown fences (```json … ```) will not parse. json_schema needs the optional ajv dependency, and regex / must_not_match need the optional recheck dependency (for ReDoS safety).

Example configs: code-evals applies these methods as format and policy gates on an agent's final answer; tool-call-accuracy points them at a tool call instead — json_schema over the arguments, must_not_match over the result — with an LLM judge alongside for argument fidelity.

Custom evaluators

You can create a custom judge metric with dt-evals evaluators add, then enable it in metrics.enabled like any built-in evaluator.

Example custom evaluator definition (created by the interactive wizard and stored locally):

{
  "id": "answer-style",
  "version": "1",
  "name": "Answer Style",
  "description": "Checks whether the answer is concise, direct, and well structured.",
  "prompt": "Evaluate the answer style for this request. User request: {{input}} Answer: {{output}} Return a continuous score from 0.0 to 1.0.",
  "requiredFields": ["input", "output"],
  "scoring": {
    "type": "continuous",
    "range": [0, 1],
    "threshold": 0.7
  }
}

Enable it in your eval config:

metrics:
  enabled:
    - faithfulness
    - answer-style

Useful commands:

dt-evals evaluators add
dt-evals evaluators list
dt-evals evaluators show answer-style
dt-evals evaluators test answer-style

Evaluation levels (agent trajectory)

By default every span is scored on its own (scope.level: agent-span) — one verdict per step. Set scope.level: agent-session to evaluate the whole agent trajectory instead: spans are grouped into one conversation (by gen_ai.conversation.id, falling back to trace.id) and a single judge call runs over the joined transcript.

Completion, recovery, and coherence only exist across steps — score each span alone and they are invisible. Any evaluator, built-in or custom, can run at either level; at session level keepPartTypes, maxMessages, and maxConversations shape the transcript.

Example configs: agent-session judges whole-conversation completeness; session-routing-accuracy lifts routing accuracy to the trajectory level.

Drift detection

drift compares current score distributions against a 7 day baseline of prior evaluation events.

Use it when you care about regressions that show up gradually, not just single-run threshold breaches.

Examples

Runnable, documented example configs live under examples/. Each folder has its own README explaining when to use it and how to adapt it.

TypeScript Library

The repository also includes dt-eval-lib — a standalone TypeScript package for running the same judge-based metrics directly in code, tests, and CI pipelines without the CLI. It supports all six providers and the full built-in evaluator catalog.

PII Handling

For CLI trace evaluation runs, the following are masked in memory before content is sent to an external judge model:

• email addresses
• phone numbers
• credit card numbers
• social security numbers

The original values are not sent to the judge by the CLI path.

Data Expectations

The CLI reads OTel GenAI semconv fields by default:

This makes the CLI a good fit for apps instrumented with OpenTelemetry GenAI conventions. For other tracing schemas — including OpenInference — use scope.spanFields to map their attributes to the canonical fields above. OpenInference commonly uses attributes such as llm.input_messages, llm.output_messages, input.value, output.value, and llm.model_name. A small set of legacy fallback attributes (gen_ai.prompt.<n>.content/role, gen_ai.completion.0.content) is also probed for compatibility with older emitters. See dynatrace-ai-agent-instrumentation-examples for example instrumentations.

Local Development

# from the repo root — install all workspace dependencies
npm install

npm run build --workspace=dt-eval-lib
npm run build --workspace=dt-eval-cli

Run locally without a build:

npm run dev -- configure
npm run dev -- run --since 1h --dry-run

Run tests:

npm test

Contributing

Issues and pull requests are welcome.

If you want to work on a larger change, open an issue first so the direction is clear before implementation starts.

License

Apache 2.0