Repository navigation
Commit 76027dd
feat(evaluation): add optional eval set result persistence to AgentEvaluator
Merge google#4414
**Please ensure you have read the [contribution guide](https://github.com/google/adk-python/blob/main/CONTRIBUTING.md) before creating a pull request.**
### Link to Issue or Description of Change
**1. Link to an existing issue (if applicable):**
- Related: google#4410
- Fixes: google#2602
**Problem:**
`AgentEvaluator.evaluate()` did not support built-in eval set result persistence, making it harder to reuse the same workflow as CLI/Web paths that already use `EvalSetResultsManager`.
Also, introducing new parameters in the middle of method signatures would break positional-argument compatibility for existing users.
**Solution:**
This PR adds optional eval result persistence to `AgentEvaluator` while preserving backward compatibility:
- Add optional parameters to `AgentEvaluator.evaluate()` and `AgentEvaluator.evaluate_eval_set()`:
- `app_name: Optional[str] = None`
- `eval_set_results_manager: Optional[EvalSetResultsManager] = None`
- Persist results per eval set (a single save aggregating all `EvalCaseResult`s), aligning `AgentEvaluator` with existing CLI/Web/API (`LocalEvalService`) persistence behavior.
- Resolve `app_name` from explicit input first, then derive from `agent_module` (including `.agent` suffix handling).
- Save results before failure assertion so failed eval runs still leave artifacts for inspection.
- Keep existing positional argument behavior by appending new parameters at the end of public method signatures.
- Add/extend tests to verify:
- explicit and derived `app_name`
- save-on-failure behavior
- argument propagation from `evaluate()` to `evaluate_eval_set()`
- positional-argument backward compatibility
- Add an integration usage example for `app_name` omission with `LocalEvalSetResultsManager`.
- For multi-run evals, all runs and eval cases are aggregated into a single result file per eval set (each run contributes one `EvalCaseResult`).
### Testing Plan
**Unit Tests:**
- [x] I have added or updated unit tests for my change.
- [x] All unit tests pass locally.
```
% pytest tests/unittests/evaluation
======================== 357 passed, 169 warnings in 9.68s =========================
```
**Manual End-to-End (E2E) Tests:**
```
% pytest tests/integration/test_with_test_file.py::test_with_single_test_file_saves_eval_set_result
======================== 1 passed, 14 warnings in 5.24s ========================
```
Verify a result file is created under: `<tmp_path>/<derived_app_name>/.adk/eval_history/*.evalset_result.json` (e.g., 1 file containing 2 `EvalCaseResult`s when num_runs=2 on a single-case eval fixture).
This is helpful for debugging failed integration tests.
### Checklist
- [x] I have read the [CONTRIBUTING.md](https://github.com/google/adk-python/blob/main/CONTRIBUTING.md) document.
- [x] I have performed a self-review of my own code.
- [x] I have commented my code, particularly in hard-to-understand areas.
- [x] I have added tests that prove my fix is effective or that my feature works.
- [x] New and existing unit tests pass locally with my changes.
- [x] I have manually tested my changes end-to-end.
- [x] Any dependent changes have been merged and published in downstream modules.
### Additional context
- This PR intentionally preserves public API positional compatibility by appending new optional parameters at the tail of method signatures.
- A generated local eval result JSON file may exist in the working tree from manual verification and is intentionally not part of the code change.
Co-authored-by: Yi Liu <[email protected]>
COPYBARA_INTEGRATE_REVIEW=google#4414 from ftnext:agent-evaluator-save-evalset-result 873973e
PiperOrigin-RevId: 9625970581 parent 74e7167 commit 76027dd
3 files changed
Lines changed: 343 additions & 5 deletions
File tree
- src/google/adk/evaluation
- tests
- integration
- unittests/evaluation
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
| 57 | + | |
57 | 58 | | |
58 | 59 | | |
59 | 60 | | |
| |||
130 | 131 | | |
131 | 132 | | |
132 | 133 | | |
| 134 | + | |
| 135 | + | |
133 | 136 | | |
134 | 137 | | |
135 | 138 | | |
| |||
155 | 158 | | |
156 | 159 | | |
157 | 160 | | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
158 | 165 | | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
159 | 171 | | |
160 | 172 | | |
161 | 173 | | |
| |||
180 | 192 | | |
181 | 193 | | |
182 | 194 | | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
183 | 201 | | |
184 | 202 | | |
185 | 203 | | |
186 | 204 | | |
187 | 205 | | |
188 | 206 | | |
189 | 207 | | |
| 208 | + | |
190 | 209 | | |
191 | 210 | | |
192 | 211 | | |
| 212 | + | |
193 | 213 | | |
194 | 214 | | |
195 | 215 | | |
| |||
249 | 269 | | |
250 | 270 | | |
251 | 271 | | |
| 272 | + | |
| 273 | + | |
252 | 274 | | |
253 | 275 | | |
254 | 276 | | |
| |||
275 | 297 | | |
276 | 298 | | |
277 | 299 | | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
278 | 304 | | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
279 | 310 | | |
280 | 311 | | |
281 | 312 | | |
| |||
302 | 333 | | |
303 | 334 | | |
304 | 335 | | |
| 336 | + | |
| 337 | + | |
305 | 338 | | |
306 | 339 | | |
307 | 340 | | |
| |||
636 | 669 | | |
637 | 670 | | |
638 | 671 | | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
639 | 675 | | |
640 | 676 | | |
641 | 677 | | |
| |||
652 | 688 | | |
653 | 689 | | |
654 | 690 | | |
655 | | - | |
656 | | - | |
657 | 691 | | |
658 | 692 | | |
659 | 693 | | |
| |||
662 | 696 | | |
663 | 697 | | |
664 | 698 | | |
| 699 | + | |
665 | 700 | | |
666 | 701 | | |
667 | 702 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| 16 | + | |
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
| |||
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
0 commit comments