Breaking changes

• The eval set id is agentevals.eval.set.id everywhere. The resource attribute agentevals.eval_set_id and the evaluation result attribute agentevals.eval_set.id are both renamed to agentevals.eval.set.id. Update OTEL_RESOURCE_ATTRIBUTES in your producers (the old name is no longer read) and any queries or dashboards over gen_ai.evaluation.result records. The SDK sets the new name. (#278)

What's Changed

• Paused turns score as one turn. A turn that pauses for an approval or a question and resumes as a new invoke_agent span, often in a new trace, is evaluated whole when the resumed span links to the span that started the turn. It keeps the first part's input and the last part's answer, and its tools are counted once. Linked traces are evaluated together in every group_by mode. (#278)
• Results name the span they attach to. TurnRef gains evaluated_trace_id and evaluated_span_id, the span agentevals attaches its gen_ai.evaluation.result records to. (#278)
• Run ids on every evaluation record. EvaluateTraces takes an optional run_id (a UUID), stamped on every record of the call. Evaluations recorded in run storage now carry the stored run's id on their records on every route, and recording never writes into a run that already holds the id. (#278)
• All four gen_ai.evaluation.score.label values (pass, fail, not_evaluated, informational) are documented. (#278)

Full Changelog: v0.12.1...v0.13.0

What's Changed

• SDK no longer sends application OTLP credentials to agentevals. With OpenTelemetry 1.45, the SDK exporters picked up headers and the mTLS client certificate from the application's OTEL_EXPORTER_OTLP_* settings and sent them to the agentevals endpoint. When those certificate files were missing on the host, exports failed instead. Only transport headers are now sent, using the default trust store with no client certificate and no proxy. Upgrade if you use the SDK and set OTEL_EXPORTER_OTLP_*HEADERS or *CLIENT_CERTIFICATE for your own backend. (#277)

Other

• Python and npm dependency updates, including OpenTelemetry 1.45.2 and authlib 1.9.4. (#275, #276)
• CI checks the highest kagent version tag instead of the GitHub latest release. (#274)

Full Changelog: v0.12.0...v0.12.1

Highlights

• ADK 2.13 efficiency metrics. tool_call_count_v1, inference_call_count_v1, token_usage_v1 and invocation_duration_v1 now work. They report a value without passing or failing, shown as [INFO] in the CLI and as neutral values in the UI. They take no threshold. (#265)
• Judges see the agent's available tools. Captured tool declarations, with descriptions and parameter schemas, are passed to ADK judges, so they can evaluate tool use against what the agent could have called. (#264, thanks @erauner12)
• Fix: tool calls keep their original Agent identity during trace conversion, so native judges no longer fail on multi Agent traces. (#267, thanks @erauner12)

Upgrade notes

• google-adk minimum is now 2.13.
• protobuf upper bound moves from <7 to <8.
• The v1alpha1 EvalStatus enum gains EVAL_STATUS_INFORMATIONAL = 4. It is wire compatible, but gRPC clients that switch over every status should handle the new value.

Other

• Kubernetes example updated for kagent v1.0.0-alpha12. (#256)
• Python, npm and GitHub Actions dependency updates.

Full Changelog: v0.11.0...v0.12.0

What's Changed

• feat: v1alpha1 protobuf contract + native gRPC by @krisztianfekete in #251
• Give rubric_based_* metrics a public rubric path (#132) by @erauner12 in #229
• fix: resolve the package version from v* tags only by @krisztianfekete in #254
• chore(ui): bump dev dependencies by @krisztianfekete in #255

Full Changelog: v0.10.0...v0.11.0

agentevals v0.10.0

agentevals now reads, stores and evaluates agent telemetry as plain OpenTelemetry. Send OTLP from any GenAI instrumentation, evaluate turns against a golden run, and get the scores back into your observability pipeline as OTel events.

Highlights

• OpenTelemetry native ingestion. OTLP over HTTP (:4318, now spec compliant with gzip, proper 400 errors and protobuf responses) and gRPC (:4317). Files can be OTLP JSON or protobuf, Jaeger, JSONL, or Collector file exporter output. See Sending telemetry to agentevals.
• One conversation model everywhere. The CLI, the API, MCP, the live UI and the run worker share the same turns. A model call wrapped by a framework counts once, a delegate agent's usage counts toward its turn, and tool rounds join across traces.
• Scores as OTel events. --emit-otel (or AGENTEVALS_EVALUATION_EVENTS=true) sends every score as a gen_ai.evaluation.result log event attached to the turn it scores, so scores show up next to your traces in any backend. See OpenTelemetry pipelines.
• Live sessions. Sessions are grouped by agentevals.session_name, gen_ai.conversation.id or session.id. Reruns get their own session, retried spans are deduplicated, and a memory budget caps the store (AGENTEVALS_LIVE_MAX_BYTES). See Sessions.
• Live UI:
  • Failed turns are shown and counted.
  • Repeated calls to the same tool each show their own result.
  • Sessions are sorted by start time.
  • Sessions without captured message content are marked and can't become a golden run.
  • The empty page lists the OTLP endpoints.
• Grouping control. --group-by auto|trace|conversation decides what one evaluation covers. See Grouping traces into conversations.
• Custom evaluator protocol 1.1. Tool responses gain a structured response field; 1.0 evaluators keep working. Upgrade the SDK to use it: pip install -U agentevals-evaluator-sdk (0.2.0). See Protocol versioning.
• Security:
  • Remote evaluator refs must be plain relative paths, and servers only run refs listed in their source's index.
  • The evaluator cache is stricter.
  • The production Collector recipe keeps message content out of your backend.
  • SDK exporters never reuse your application's TLS client certificates.

Experimental: kagent 1.0 harness agents via Substrate

• Claude Code harness: tested on kagent v1.0.0-alpha9. With the Collector recipe in the Kubernetes example, you get tool calls by name and model calls with token usage. Tool arguments aren't recorded yet, so trajectory scores against a golden run with arguments will fail.
• Codex harness: not tested yet. Full support depends on kagent writing runtime GenAI spans (kagent#3121).
• Go ADK agents: full model and tool calls with arguments, results and tokens.
• The example includes a Collector filter for kagent's TaskStore spans, which keeps long turns under the per trace span limit, plus commands to send prompts from the command line.

❗Breaking changes

Ingestion

• WebSocket ingestion (/ws/traces) and AgentEvalsStreamingProcessor are removed. Send OTLP to :4318 or :4317. AgentEvals SDK sessions switch over on their own.
• Receivers reject spans and logs with invalid or all zero ids and count them in partialSuccess.

Results

• Turn boundaries and token counts follow the new model, so scores on the same telemetry can differ from v0.9.x.
• An unmatched eval case stays NOT_EVALUATED instead of falling back to the first case. The exception: an eval set with one case evaluated against one trace group is still paired. See How traces are matched to cases.
• invocation_id is now the turn's anchor span id.
• Tool response output given to custom evaluators is JSON text, no longer a Python repr. Read the structured response field instead (SDK 0.2.0).

API

• Keys inside args, arguments, response, result, output and details are no longer converted to camelCase.
• get-trace returns one OTLP JSON document in trace_content, no longer JSONL spans.
• The SSE span_received event carries span stubs (ids and name) only.

Evaluators

• The openai_eval evaluator and the [openai] extra are removed, because OpenAI shuts down its Evals API on 2026-10-31. Use response_match_score, final_response_match_v2 or a code evaluator instead. Migration 000002_drop_openai_eval deletes old openai_eval result rows.

Helm

• rbac.create: true now requires rbac.secretNames, or an explicit rbac.allowAllSecrets: true. See the chart values.

Old files

• Files and bug report bundles written by older versions that copied log content into every span are read as concatenated user text. Record a fresh session.

Upgrading

1. Point your exporters or Collector at :4318 (HTTP) or :4317 (gRPC). See Forward agent telemetry to agentevals.
2. Turn on message content capture in your instrumentation. See Producer setup for per framework settings.
3. Helm users: set rbac.secretNames or rbac.allowAllSecrets.
4. Replace any openai_eval evaluators before upgrading the database.

What's Changed

• chore(deps): bump the python group with 4 updates by @dependabot[bot] in #235
• fix: harden remote evaluators by @krisztianfekete in #237
• refactor: unship the OpenAI Evals and fix baseline migration rollback by @krisztianfekete in #238
• fix(otlp): make the OTLP/HTTP receiver spec compliant by @JHf0912 in #223
• feat: OTel-native refactor by @krisztianfekete in #241
• test: kagent compat fixtures, ordered replay, and a daily Weaver check of kagent's registry by @krisztianfekete in #245
• fix: keep GenAI content out of the backend recipe, isolate SDK TLS, set content capture by @krisztianfekete in #246
• fix: smaller release blockers by @krisztianfekete in #247

New Contributors

• @JHf0912 made their first contribution in #223

Full Changelog: v0.9.12...v0.10.0

What's Changed

• chore(deps): bump the npm group in /ui with 10 updates by @dependabot[bot] in #217
• ci: hold mcp at 1.x in the dependabot python group by @krisztianfekete in #219
• chore(deps): bump astral-sh/setup-uv from 10.0.1 to 10.1.0 in the github-actions group by @dependabot[bot] in #221
• chore(deps): bump lucide-react from 1.43.0 to 1.44.0 in /ui in the npm group by @dependabot[bot] in #220
• chore(deps): bump the python group with 4 updates by @dependabot[bot] in #222
• chore(deps): bump the python group with 4 updates by @dependabot[bot] in #227
• chore(deps): bump the github-actions group with 3 updates by @dependabot[bot] in #226
• chore(deps): bump the npm group in /ui with 5 updates by @dependabot[bot] in #225
• ci: run ruff check with --no-fix so autofixable violations fail by @rangarajan19 in #224
• ci: run integration tests in CI by @Oscar-Williams in #228
• chore(deps): bump astral-sh/setup-uv from 10.1.0 to 10.2.0 in the github-actions group by @dependabot[bot] in #231
• chore(deps): bump the python group with 4 updates by @dependabot[bot] in #233
• chore(deps): bump the npm group across 1 directory with 7 updates by @dependabot[bot] in #232
• fix: feed tool evidence to hallucinations_v1 by @krisztianfekete in #234

New Contributors

• @rangarajan19 made their first contribution in #224
• @Oscar-Williams made their first contribution in #228

Full Changelog: v0.9.11...v0.9.12

What's Changed

• chore(deps): bump the python group with 4 updates by @dependabot[bot] in #215
• chore(deps): bump the github-actions group across 1 directory with 2 updates by @dependabot[bot] in #213
• chore(deps): bump the npm group across 1 directory with 7 updates by @dependabot[bot] in #214
• Migrate base image to chainguard + python package by @inFocus7 in #216

Full Changelog: v0.9.10...v0.9.11

What's Changed

• fix(otel): share one AnyValue decoder across all OTLP paths by @Leroyyyyyyyyy in #187
• chore(deps): bump the python group with 2 updates by @dependabot[bot] in #211
• chore(deps): bump the npm group across 1 directory with 7 updates by @dependabot[bot] in #210
• chore(deps): bump node from 25-bookworm-slim to 26-bookworm-slim in the docker-base-images group across 1 directory by @dependabot[bot] in #209
• ci: build and smoke test the docker image on every PR by @krisztianfekete in #212

New Contributors

• @Leroyyyyyyyyy made their first contribution in #187

Full Changelog: v0.9.9...v0.9.10

What's Changed

• Security: Bump Google ADK (and others) to resolve CVEs by @inFocus7 in #188
• chore(ci): group dependabot updates and pin actions to SHAs by @krisztianfekete in #196
• fix(ci): drop unsupported semver-major-days from docker cooldown by @krisztianfekete in #197
• ci: build the UI on every PR by @krisztianfekete in #201
• chore(deps): bump the python group with 11 updates by @dependabot[bot] in #200
• chore(deps): bump the github-actions group across 1 directory with 4 updates by @dependabot[bot] in #198
• fix(ci): ignore majors for packages coupled to vite and typescript by @krisztianfekete in #202
• chore(deps): bump the npm group across 1 directory with 17 updates by @dependabot[bot] in #203

New Contributors

• @dependabot[bot] made their first contribution in #200

Full Changelog: v0.9.8...v0.9.9

What's Changed

• fix(streaming): normalize timestamps across common units by @JuSe123456 in #179
• feat(log-enrichment): add custom session ID attribute for enriched spans by @AaronProbha18 in #181
• ci: run tests with all extras and gate the lock by @krisztianfekete in #184
• chore(security): target dependency lock for CVE fixes by @inFocus7 in #182

New Contributors

• @JuSe123456 made their first contribution in #179
• @inFocus7 made their first contribution in #182

Full Changelog: v0.9.7...v0.9.8