Repository navigation
Pass captured tool definitions to ADK judges as app_details #263
Description
Problem or motivation
ADK judges read the agent's tool declarations from Invocation.app_details. rubric_based_tool_use_quality_v1, for example, falls back to "Agent has no tools." when it is absent. #234 improved this by synthesizing app_details in _enrich_app_details, but only from tool names the agent actually called:
That leaves the judge without:
- tools that were available but not called (needed to judge "should have used X" or "picked the wrong tool")
- descriptions and parameter schemas (needed to judge whether arguments were sensible)
Meanwhile, the full definitions are already extracted. genai/overlay.py reads them from ADK's LLM request into gen_ai.tool.definitions, and extract.py stores them on each LlmCall.tool_definitions:
- https://github.com/agentevals-dev/agentevals/blob/618c942/src/agentevals/genai/overlay.py#L231
- https://github.com/agentevals-dev/agentevals/blob/618c942/src/agentevals/genai/extract.py#L318
to_adk_invocation never uses them, so callers that build invocations with it and call get_evaluator(...).evaluate_invocations(...) directly get no declarations at all, not even names:
Proposed solution
In to_adk_invocation, build app_details from turn.llm_calls[*].tool_definitions: deduplicate by name, then map name, description and parameters to FunctionDeclaration(name=..., description=..., parameters_json_schema=...).
Keep _enrich_app_details as the fallback for traces that carry no definitions (content capture off, or non-ADK sources that don't emit gen_ai.tool.definitions). It already skips invocations that have app_details.
Tests: an ADK trace with an available-but-uncalled tool should show that tool, its description and its schema in the tool-use judge prompt; a trace without definitions should keep today's names-only behavior.
Alternatives considered
Callers can build app_details themselves from Turn.llm_calls. That works, but every caller would duplicate the same mapping, and the built-in evaluation path would still only have names.
Additional context
We use rubric_based_tool_use_quality_v1 to replace hand-maintained tool allowlists with rubrics such as "gathered evidence relevant to the reported workload". That only works if the judge can see which tools were available and what they do. Happy to open the PR.
Human confirmation
- I am a human (not a bot, agent, or AI) filing this issue.
Activity