Skip to content

Commit 76027dd

Browse files
ftnexti-yliu
authored andcommitted
feat(evaluation): add optional eval set result persistence to AgentEvaluator
Merge google#4414 **Please ensure you have read the [contribution guide](https://github.com/google/adk-python/blob/main/CONTRIBUTING.md) before creating a pull request.** ### Link to Issue or Description of Change **1. Link to an existing issue (if applicable):** - Related: google#4410 - Fixes: google#2602 **Problem:** `AgentEvaluator.evaluate()` did not support built-in eval set result persistence, making it harder to reuse the same workflow as CLI/Web paths that already use `EvalSetResultsManager`. Also, introducing new parameters in the middle of method signatures would break positional-argument compatibility for existing users. **Solution:** This PR adds optional eval result persistence to `AgentEvaluator` while preserving backward compatibility: - Add optional parameters to `AgentEvaluator.evaluate()` and `AgentEvaluator.evaluate_eval_set()`: - `app_name: Optional[str] = None` - `eval_set_results_manager: Optional[EvalSetResultsManager] = None` - Persist results per eval set (a single save aggregating all `EvalCaseResult`s), aligning `AgentEvaluator` with existing CLI/Web/API (`LocalEvalService`) persistence behavior. - Resolve `app_name` from explicit input first, then derive from `agent_module` (including `.agent` suffix handling). - Save results before failure assertion so failed eval runs still leave artifacts for inspection. - Keep existing positional argument behavior by appending new parameters at the end of public method signatures. - Add/extend tests to verify: - explicit and derived `app_name` - save-on-failure behavior - argument propagation from `evaluate()` to `evaluate_eval_set()` - positional-argument backward compatibility - Add an integration usage example for `app_name` omission with `LocalEvalSetResultsManager`. - For multi-run evals, all runs and eval cases are aggregated into a single result file per eval set (each run contributes one `EvalCaseResult`). ### Testing Plan **Unit Tests:** - [x] I have added or updated unit tests for my change. - [x] All unit tests pass locally. ``` % pytest tests/unittests/evaluation ======================== 357 passed, 169 warnings in 9.68s ========================= ``` **Manual End-to-End (E2E) Tests:** ``` % pytest tests/integration/test_with_test_file.py::test_with_single_test_file_saves_eval_set_result ======================== 1 passed, 14 warnings in 5.24s ======================== ``` Verify a result file is created under: `<tmp_path>/<derived_app_name>/.adk/eval_history/*.evalset_result.json` (e.g., 1 file containing 2 `EvalCaseResult`s when num_runs=2 on a single-case eval fixture). This is helpful for debugging failed integration tests. ### Checklist - [x] I have read the [CONTRIBUTING.md](https://github.com/google/adk-python/blob/main/CONTRIBUTING.md) document. - [x] I have performed a self-review of my own code. - [x] I have commented my code, particularly in hard-to-understand areas. - [x] I have added tests that prove my fix is effective or that my feature works. - [x] New and existing unit tests pass locally with my changes. - [x] I have manually tested my changes end-to-end. - [x] Any dependent changes have been merged and published in downstream modules. ### Additional context - This PR intentionally preserves public API positional compatibility by appending new optional parameters at the tail of method signatures. - A generated local eval result JSON file may exist in the working tree from manual verification and is intentionally not part of the code change. Co-authored-by: Yi Liu <[email protected]> COPYBARA_INTEGRATE_REVIEW=google#4414 from ftnext:agent-evaluator-save-evalset-result 873973e PiperOrigin-RevId: 962597058
1 parent 74e7167 commit 76027dd

3 files changed

Lines changed: 343 additions & 5 deletions

File tree

‎src/google/adk/evaluation/agent_evaluator.py‎

Lines changed: 37 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@
5454
from .eval_metrics import PrebuiltMetrics
5555
from .eval_result import EvalCaseResult
5656
from .eval_set import EvalSet
57+
from .eval_set_results_manager import EvalSetResultsManager
5758
from .eval_sets_manager import EvalSetsManager
5859
from .evaluator import EvalStatus
5960
from .in_memory_eval_sets_manager import InMemoryEvalSetsManager
@@ -130,6 +131,8 @@ async def evaluate_eval_set(
130131
print_detailed_results: bool = True,
131132
artifact_service: Optional[BaseArtifactService] = None,
132133
output_file: Optional[str] = None,
134+
app_name: Optional[str] = None,
135+
eval_set_results_manager: Optional[EvalSetResultsManager] = None,
133136
) -> None:
134137
"""Evaluates an agent using the given EvalSet.
135138
@@ -155,7 +158,16 @@ async def evaluate_eval_set(
155158
passing and failing metrics) are written to this path as a CSV file.
156159
Disabled by default. The parent directory is created if it does not
157160
already exist.
161+
app_name: The application name used by eval set results manager while
162+
persisting eval set results.
163+
eval_set_results_manager: Optional manager used to persist the eval set
164+
evaluation result as `*.evalset_result.json`.
158165
"""
166+
if eval_set_results_manager is not None and not app_name:
167+
raise ValueError(
168+
"app_name is required when eval_set_results_manager is provided."
169+
)
170+
159171
if criteria:
160172
logger.warning(
161173
"`criteria` field is deprecated and will be removed in future"
@@ -180,16 +192,24 @@ async def evaluate_eval_set(
180192
)
181193
live_model_config = eval_config.live_model_config
182194

195+
# `eval_set_results_manager`, when provided, is what persists the eval
196+
# results as `*.evalset_result.json` files (via LocalEvalService), stored
197+
# under `app_name`. When no manager is given, nothing is saved, so a dummy
198+
# `app_name` is fine here.
199+
app_name = app_name or "test_app"
200+
183201
# Step 1: Perform evals, basically inferencing and evaluation of metrics
184202
eval_results_by_eval_id = await AgentEvaluator._get_eval_results_by_eval_id(
185203
agent_for_eval=agent_for_eval,
186204
eval_set=eval_set,
187205
eval_metrics=eval_metrics,
188206
num_runs=num_runs,
189207
user_simulator_provider=user_simulator_provider,
208+
app_name=app_name,
190209
live_model_config=live_model_config,
191210
artifact_service=artifact_service,
192211
app=app,
212+
eval_set_results_manager=eval_set_results_manager,
193213
)
194214

195215
# Step 2: Post-process the results!
@@ -249,6 +269,8 @@ async def evaluate(
249269
print_detailed_results: bool = True,
250270
artifact_service: Optional[BaseArtifactService] = None,
251271
output_file: Optional[str] = None,
272+
app_name: Optional[str] = None,
273+
eval_set_results_manager: Optional[EvalSetResultsManager] = None,
252274
) -> None:
253275
"""Evaluates an Agent given eval data.
254276
@@ -275,7 +297,16 @@ async def evaluate(
275297
this path as a CSV file. Disabled by default. When the eval data spans
276298
multiple test files, results from all of them are appended to the same
277299
file.
300+
app_name: The application name used by eval set results manager while
301+
persisting eval set results.
302+
eval_set_results_manager: Optional manager used to persist the eval set
303+
evaluation result as `*.evalset_result.json`.
278304
"""
305+
if eval_set_results_manager is not None and not app_name:
306+
raise ValueError(
307+
"app_name is required when eval_set_results_manager is provided."
308+
)
309+
279310
test_files = []
280311
if isinstance(eval_dataset_file_path_or_dir, str) and os.path.isdir(
281312
eval_dataset_file_path_or_dir
@@ -302,6 +333,8 @@ async def evaluate(
302333
num_runs=num_runs,
303334
agent_name=agent_name,
304335
print_detailed_results=print_detailed_results,
336+
app_name=app_name,
337+
eval_set_results_manager=eval_set_results_manager,
305338
artifact_service=artifact_service,
306339
output_file=output_file,
307340
)
@@ -636,6 +669,9 @@ async def _get_eval_results_by_eval_id(
636669
live_model_config: Optional[LiveModelConfig] = None,
637670
artifact_service: Optional[BaseArtifactService] = None,
638671
app: Optional[App] = None,
672+
*,
673+
app_name: str,
674+
eval_set_results_manager: Optional[EvalSetResultsManager] = None,
639675
) -> dict[str, list[EvalCaseResult]]:
640676
"""Returns EvalCaseResults grouped by eval case id.
641677
@@ -652,8 +688,6 @@ async def _get_eval_results_by_eval_id(
652688
except ModuleNotFoundError as e:
653689
raise ModuleNotFoundError(MISSING_EVAL_DEPENDENCIES_MESSAGE) from e
654690

655-
# It is okay to pick up this dummy name.
656-
app_name = "test_app"
657691
eval_service = LocalEvalService(
658692
root_agent=agent_for_eval,
659693
eval_sets_manager=AgentEvaluator._get_eval_sets_manager(
@@ -662,6 +696,7 @@ async def _get_eval_results_by_eval_id(
662696
user_simulator_provider=user_simulator_provider,
663697
artifact_service=artifact_service,
664698
app=app,
699+
eval_set_results_manager=eval_set_results_manager,
665700
)
666701

667702
if live_model_config:

‎tests/integration/test_with_test_file.py‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
# limitations under the License.
1414

1515
from google.adk.evaluation.agent_evaluator import AgentEvaluator
16+
from google.adk.evaluation.local_eval_set_results_manager import LocalEvalSetResultsManager
1617
import pytest
1718

1819

@@ -35,3 +36,40 @@ async def test_with_folder_of_test_files_long_running():
3536
),
3637
num_runs=4,
3738
)
39+
40+
41+
@pytest.mark.asyncio
42+
async def test_with_single_test_file_saves_eval_set_result(
43+
tmp_path,
44+
):
45+
"""Persists eval set results under the explicitly provided app_name."""
46+
eval_set_results_manager = LocalEvalSetResultsManager(
47+
agents_dir=str(tmp_path)
48+
)
49+
await AgentEvaluator.evaluate(
50+
agent_module="tests.integration.fixture.home_automation_agent",
51+
eval_dataset_file_path_or_dir=(
52+
"tests/integration/fixture/home_automation_agent/simple_test.test.json"
53+
),
54+
num_runs=2,
55+
app_name="home_automation_agent",
56+
eval_set_results_manager=eval_set_results_manager,
57+
)
58+
59+
# Results are aggregated into a single eval set result file (matching
60+
# LocalEvalService), containing one EvalCaseResult per run.
61+
saved_result_files = list(
62+
(tmp_path / "home_automation_agent" / ".adk" / "eval_history").glob(
63+
"*.evalset_result.json"
64+
)
65+
)
66+
assert len(saved_result_files) == 1
67+
68+
saved_result_ids = eval_set_results_manager.list_eval_set_results(
69+
"home_automation_agent"
70+
)
71+
assert len(saved_result_ids) == 1
72+
eval_set_result = eval_set_results_manager.get_eval_set_result(
73+
"home_automation_agent", saved_result_ids[0]
74+
)
75+
assert len(eval_set_result.eval_case_results) == 2

0 commit comments

Comments
 (0)