agentevals reads standard OpenTelemetry GenAI telemetry: spans, plus log events for message content. Any producer that follows the GenAI semantic conventions works. This page says what agentevals needs, how to set up common producers, and what the receiver accepts.
| To get | Send |
|---|---|
| Turns | An invoke_agent or invoke_workflow span per user turn. Without one, each root span with model calls becomes a turn. |
| User input and final answer | gen_ai.input.messages / gen_ai.output.messages on the agent span or the model call spans (as attributes or as log events). |
| Tool trajectory | execute_tool spans with gen_ai.tool.name, ideally gen_ai.tool.call.id. |
| Tool arguments and results | gen_ai.tool.call.arguments / gen_ai.tool.call.result, or tool_call / tool_call_response message parts. |
| Tokens and model | gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.request.model on model call spans. |
| Errors | Span status ERROR and error.type. |
| Sessions | One of the keys in Sessions. |
Message content is optional. Without it you still get tools, tokens, errors and latency, but text based metrics have nothing to score.
| Producer | Set |
|---|---|
| Google ADK (Python) | Nothing for the defaults. For the latest conventions: ADK_TELEMETRY_SCHEMA_VERSION_OPT_IN=2 and OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental, with OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY (or EVENT_ONLY plus a log exporter). |
| adk-go | Nothing. The root invoke_agent span marks the turn. |
Official OpenTelemetry GenAI packages (opentelemetry-instrumentation-genai-*, 1.2b0+) |
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_AND_EVENT for content. They always use the latest conventions. |
opentelemetry-instrumentation-openai-v2 (LangChain, plain OpenAI) |
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true, and export logs too: content arrives as log events. |
| Strands | OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental. |
| OpenAI Agents SDK | OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_and_event and OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental. Tool spans carry no call id, so tools match by name. |
kagent (OpenLLMetry gen_ai.prompt.N.* attributes) |
Nothing. These are read as a legacy fallback. |
Working setups for each are in examples/zero-code-examples.
If a framework only instruments the model client (each model call is its own trace), agentevals joins a call that answers a tool result to the turn before it. This needs message content.
Live traces are grouped into sessions by the first key present:
- resource
agentevals.session_name gen_ai.conversation.id(span, then resource)session.id(span, then resource)
Traces with none of these become a session of their own when they contain GenAI spans. Traces with no GenAI span at all are held back for two minutes and shown only if a GenAI span or a key arrives for them.
Reruns with the same agentevals.session_name become name-2, name-3. A rerun is recognized by a different service.instance.id, so set one per process:
export OTEL_RESOURCE_ATTRIBUTES="agentevals.session_name=my-agent,service.instance.id=$(uuidgen)"Without service.instance.id, a new trace counts as a rerun once the previous session has been finished for 30 seconds. Sessions keyed by a conversation or session id never split.
A trace is finished 3 seconds after its root span arrives (30 seconds if no root ever arrives), and a session when all its traces are. New spans reopen a finished session; logs that arrive late update it without reopening. Finished sessions stay in memory for 2 hours.
Anyone who can reach the receiver can add traces to a session by reusing its key, and the server has no authentication. Bind it to 127.0.0.1 (agentevals serve --host 127.0.0.1) on shared machines, and do not expose it to untrusted networks.
In the UI, pick a golden session and evaluate the others against it; each session is scored as one conversation. From scripts:
| Endpoint | Use |
|---|---|
GET /api/streaming/sessions |
List sessions with their turns |
GET /api/streaming/sessions/{id}/otlp |
The session as an OTLP/JSON document; agentevals run reads it back |
POST /api/streaming/create-eval-set |
Build an eval set from a session |
POST /api/streaming/evaluate-sessions |
Score every finished session against a golden session |
agentevals needs whole traces. Keep the SDK sampler at its default (always on). If you sample in a Collector, sample whole traces (tail sampling) and keep the logs of the traces you keep. A trace with missing spans gives missing turns or tools.
- Span names. Spans are classified by
gen_ai.operation.name. schema_url.- Log records that are not
gen_ai.*events, and logs without trace context. - Its own
gen_ai.evaluation.resultevents, so a pipeline that loops back does not feed results in again.
Vendor attributes (gcp.vertex.agent.*) are read as a fallback, never required.
| Port | Protocol |
|---|---|
| 4318 | OTLP/HTTP: /v1/traces, /v1/logs, JSON or protobuf, gzip or no compression |
| 4317 | OTLP/gRPC: trace and log export, gzip |
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
# or, for gRPC
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpcTrace and span ids must be valid hex (32 and 16 characters). Records with bad ids are rejected one by one.
| Situation | Response |
|---|---|
| Everything accepted | 200, no partialSuccess |
| Some records refused for good (bad ids, a trace or session limit) | 200 with partialSuccess counts and a reason. Do not retry. |
| Any record refused because the server is full (memory, session slots, log buffer) | 503 with Retry-After (gRPC UNAVAILABLE). Retry the whole export; records already stored are not stored twice. |
| Body cannot be decoded | 400 |
Unsupported Content-Type or Content-Encoding |
415. Use compression: gzip or none. |
| Body over 64 MiB, compressed body over 8 MiB, or more than 32 gzip members | 413 / 400 |
Error bodies are google.rpc.Status messages in the request's content type.
| Limit | Value |
|---|---|
| Spans per trace / per session | 10,000 / 50,000 |
| Logs per trace / per session | 5,000 / 20,000 |
| Traces per session | 1,000 |
| Sessions | 100 (the oldest finished session is evicted first) |
| Memory | 1 GiB of decoded telemetry (AGENTEVALS_LIVE_MAX_BYTES) |
| Finished session kept for | 2 hours |
| Logs that arrive before their spans | kept for 5 minutes |
A Collector sending large batches should cap batch size so a request stays under 64 MiB.