GenAI revamp: OTel native trace ingestion and evaluation #236

👋
I'm starting a larger refactor of how agentevals ingests and evaluates GenAI traces. I'm opening this early to share where it's going and to get feedback before the bigger changes land. This should also help with in-flight PRs.

Why

• Internally, agentevals uses ADK's Invocation / EvalSet as its data model. It also picks one format for the whole trace, and ADK wins whenever its markers show up. That breaks on traces that mix instrumentations, and on newer producers. Today, some adk-python telemetry settings, adk-go traces, and ADK's announced telemetry changes all produce 0 invocations.
• The GenAI semantic conventions moved to their own repo (semantic-conventions-genai). They have no release yet and still make breaking renames, so we need a sturdier way to keep up with them.

Direction

• A lossless span model that keeps resource, scope, kind, status, events, links and correlated logs
• Small adapters for each instrumentation dialect (ADK, OpenInference, OpenLLMetry, Strands, potentially others), applied span by span, all feeding one semconv extractor
• A neutral Conversation / Turn / Step model keyed by real span ids
• Metrics implemented in agentevals itself. ADK becomes an optional agentevals[adk] extra, and ADK EvalSets stay importable
• Evaluation results also emitted as standard gen_ai.evaluation.result log records
• Semconv constants generated from a pinned upstream commit instead of maintained by hand

Rollout

Planning to do the work in only a few PRs, maybe as few as a single one, then release it as likely v0.10.0.
There will be some breaking changes such as retiring WebSocket ingestion or changing the golden format, making ADK an optional extra, and invocations will become turns. Unless we have multiple user's reporting the need for a deprecation cycle, there likely won't be one. Let us know!

Feedback welcome

Which producers and instrumentations do you rely on? What would this break for you? What would you like to see prioritized?