Repository navigation
feat: join resumed turns, expose the evaluated span and run id, rename the eval set id - #278
Merged
Merged
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Concurrent evaluations can claim the same run ID and overwrite or mix results because the ID is not atomically reserved before records are emitted.
5 open issues
2 files not reviewed
gen/go/agentevals/api/v1alpha1/evaluation.pb.go: Generated filesrc/agentevals/api/v1alpha1/evaluation_pb2.py: Generated file
What changed in this PR
This PR improves how agentevals evaluates paused and resumed agent turns and correlates evaluation results with telemetry.
Changes:
- Joins linked turn segments across traces and deduplicates repeated tool calls.
- Exposes evaluated span IDs and propagates run IDs through evaluation and storage.
- Renames the eval-set telemetry attribute to
agentevals.eval.set.idand updates documentation, examples, generated bindings, and tests.
| File | Description |
|---|---|
ui/src/lib/types.ts |
Adds evaluated span identifiers to turn references. |
ui/src/generated/agentevals/api/v1alpha1/evaluation_pb.js |
Regenerates descriptors for new API fields. |
ui/src/generated/agentevals/api/v1alpha1/evaluation_pb.d.ts |
Adds generated types for span and run IDs. |
tests/test_sdk.py |
Updates SDK resource-attribute assertions. |
tests/test_otlp_receiver.py |
Updates receiver fixtures and assertions for the renamed attribute. |
tests/otel/test_emit.py |
Tests labels, attribute names, and evaluated span references. |
tests/otel/builders.py |
Adds span-link support to test builders. |
tests/integration/test_live_agents.py |
Uses the renamed attribute in agent environments. |
tests/integration/conftest.py |
Updates integration trace resource attributes. |
tests/genai/test_resumed_turns.py |
Tests resumed-turn merging and trace grouping. |
tests/api/test_grpc_api.py |
Tests requested run IDs, validation, and emitted records. |
tests/api/test_evaluate_persistence.py |
Tests stored run IDs on emitted records. |
src/agentevals/runner.py |
Exposes evaluated span references and forwards run IDs. |
src/agentevals/run/service.py |
Accepts a preselected ID when recording evaluations. |
src/agentevals/otel/sdk_export.py |
Updates the attribute name in module documentation. |
src/agentevals/otel/identity.py |
Renames the shared eval-set attribute constant. |
src/agentevals/genai/grouping.py |
Groups traces connected by resume links. |
src/agentevals/genai/extract.py |
Joins resumed turns and deduplicates tool calls. |
src/agentevals/evaluation_events.py |
Shares evaluated-span selection and aligns attribute names. |
src/agentevals/api/v1alpha1/evaluation_pb2.pyi |
Regenerates Python type stubs. |
src/agentevals/api/v1alpha1/evaluation_pb2.py |
Regenerates Python protobuf descriptors. |
src/agentevals/api/rpc/services.py |
Validates and forwards caller-supplied run IDs. |
src/agentevals/api/rpc/convert.py |
Serializes expanded turn references. |
src/agentevals/api/routes.py |
Chooses storage-backed run IDs before evaluation. |
src/agentevals/api/evaluation.py |
Carries run IDs through streaming evaluation and persistence. |
README.md |
Documents the renamed eval-set attribute. |
proto/agentevals/api/v1alpha1/evaluation.proto |
Defines evaluated span fields and optional request run ID. |
gen/go/agentevals/api/v1alpha1/evaluation.pb.go |
Regenerates Go protobuf bindings. |
examples/zero-code-examples/strands/run.py |
Updates the Strands resource attribute. |
examples/zero-code-examples/pydantic-ai/run.py |
Updates the Pydantic AI resource attribute. |
examples/zero-code-examples/openai-agents/run.py |
Updates the OpenAI Agents resource attribute. |
examples/zero-code-examples/ollama/run.py |
Updates the Ollama resource attribute. |
examples/zero-code-examples/langchain/run.py |
Updates the LangChain resource attribute. |
examples/zero-code-examples/adk/run.py |
Updates the ADK resource attribute. |
examples/README.md |
Updates telemetry setup instructions. |
examples/kubernetes/README.md |
Updates the sample evaluation event attribute. |
docs/otel-compatibility.md |
Documents resumed-turn telemetry requirements. |
docs/opentelemetry-pipeline.md |
Documents evaluated spans, run IDs, and all score labels. |
docs/grpc-api.md |
Explains evaluated span references and request run IDs. |
docs/eval-set-format.md |
Documents grouping behavior for resumed traces. |
Review effort: Balanced
…calls across segments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A turn that pauses for an approval or a question and resumes as a new
invoke_agentspan, often in a new trace, is now scored as one turn. The resumed span must link to the agent span that started the turn, both must carry the samegen_ai.agent.name, and the resumed span must record no input of its own. The turn keeps the first part's user input and the last part's answer, and takes tools from both. A tool call recorded in both parts (same call id, tool and agent) counts once: the span that finished wins and keeps arguments only the paused span recorded. Linked traces are evaluated together in everygroup_bymode. Before, each part was its own turn and the tool was counted twice.TurnRefgainsevaluated_trace_idandevaluated_span_id: the span the turn'sgen_ai.evaluation.resultrecords attach to. They come from the same function the emitter uses, so API clients can attach results exactly where agentevals does.EvaluateTracesRequestgains an optionalrun_id(a UUID), stamped asagentevals.eval.run.idon every record of the call. With run storage, the evaluation is recorded under it, and an id that already exists is rejected withALREADY_EXISTS. Evaluations recorded in run storage now choose their run id before evaluating, so their records carry the stored run's id on every route, including/api/streaming/evaluate-sessions. Before, those records had no run id. Recording never writes into a run that another run already holds: if the id is taken by then (two calls racing with the samerun_id, say), the evaluation is not recorded and the existing run is left untouched.All four
gen_ai.evaluation.score.labelvalues are documented (pass,fail,not_evaluated,informational).Breaking changes
agentevals.eval_set_idis nowagentevals.eval.set.id. UpdateOTEL_RESOURCE_ATTRIBUTES; the old name is no longer read. The SDK sets the new name.agentevals.eval.set.idinstead ofagentevals.eval_set.id. Update queries and dashboards.Not in this PR
Result.span_idstill names the turn's anchor span.agentevals.*attributes.Testing
2067 unit tests and 29 integration tests pass. Each commit passes on its own. Ruff,
uv lock --check, buf lint, buf breaking, the Go build and vet, and the UI type check, lint and build all pass. New tests cover resumed turns and grouping, the evaluated span against the emitted record's parent, run ids on gRPC and REST, and run id conflicts on both the memory and Postgres repositories (the storage tests pass against Postgres 16).🤖 Generated with Claude Code