Skip to content

Commit 7f3517e

Browse files
i-yliucopybara-github
i-yliu
authored and
copybara-github
committed
feat(eval): run each eval case N times in adk eval for steadier verdicts
**Problem:** An agent can answer the same eval case differently from one run to the next, so a single `adk eval` run can pass or fail by chance. `adk eval` had no way to repeat eval cases, so users scripted external loops. **Solution:** - Add `EvalConfig.multi_run_config` (`MultiRunConfig`) and an `adk eval --num_runs N` flag. The flag overrides `multi_run_config.num_runs`; if neither is set, each eval case runs once. - `InferenceConfig.num_runs` makes `LocalEvalService.perform_inference` run each eval case N times under its existing parallelism limit, each run in a fresh session. A session pinned by the eval case is not reused, and a warning says so. - Every run is kept. `InferenceResult` and `EvalCaseResult` gain `repetition_index`, the 0-based index of the run, and all runs are saved to the eval history. `AgentEvaluator` labels its runs the same way. - `adk eval` judges each eval case on all of its runs. By default each metric's per-invocation scores are pooled across runs and their mean is compared to the threshold, which is how `AgentEvaluator` already judges repeated runs; a run scored only as a whole counts once, with its overall score. As in `AgentEvaluator`, whether higher or lower scores pass is inferred from the evaluator's own verdicts. A run that crashed before it could be scored is left out of every count, and fails the case unless `max_crashed_runs` allows it. - `multi_run_config` also chooses how the runs are combined. Each setting is named for working across the runs of a case, which is all they do: - `cross_run_scoring`, and `cross_run_scoring_by_metric` for individual metrics, name a method that combines scores (`mean_over_invocations`, the default, `mean_of_runs` or `min_over_runs`), judged against the metric's threshold in `criteria`, or give a rule such as `{"method": "pass_rate", "threshold": 0.9}` that counts the runs in which the metric passed. - `cross_run_criteria` adds such rules for the case as a whole. A run counts as passed when it passed with every metric evaluated, and the case fails if any criterion fails. - `custom_methods` declares methods of your own: Python functions referenced by `CodeConfig`, as custom metrics are, each with `uses_scores` saying whether it combines scores or counts passing runs. The rules name them as they name built-in methods. A method receives one `RunResult` per run, now exported from `google.adk.evaluation`, and returns a number or `None`. `adk eval` imports them before it runs any eval case, so a path that does not import fails at once. - A rule left without a value, for example because no run passed or failed, is not evaluated and is skipped, and the output says why. - `max_crashed_runs`, 0 by default, is how many crashed runs a case tolerates. A crashed run has no verdict, so it is left out of every count of passing runs, and a case whose runs all crashed never passes. - With `--print_detailed_results`, each case's verdict across its runs is printed first, with how many runs passed, each metric's method, value and threshold, and each cross-run criterion. Each of its runs follows. - With the default of one run and no rule that counts passing runs, verdicts and detailed results print as before. Co-authored-by: ftnext <[email protected]> Co-authored-by: Haran Rajkumar <[email protected]> Co-authored-by: Yi Liu <[email protected]> PiperOrigin-RevId: 1005364760
1 parent 1fbe9e0 commit 7f3517e

21 files changed

Lines changed: 5591 additions & 29 deletions

‎docs/guides/evaluation/agent_evaluator/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -217,7 +217,7 @@ the manager without it raises `ValueError` before anything runs.
217217
`LocalEvalSetResultsManager(agents_dir=...)` writes a single
218218
`*.evalset_result.json` per evaluated eval set under
219219
`<agents_dir>/<app_name>/.adk/eval_history/`, holding one `EvalCaseResult` for
220-
each of the `num_runs` runs.
220+
each of the `num_runs` runs, labeled with its 0-based `repetition_index`.
221221

222222
`output_file` is the flat alternative: one CSV row per metric per invocation,
223223
carrying the threshold, the score, the status, the prompt, and both the expected

‎docs/guides/evaluation/eval_config/index.md‎

Lines changed: 254 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,8 @@ metric has options you want to set:
7878
```
7979

8080
`threshold` is required in the object form. The metric passes when the mean of
81-
its per-invocation scores, across every run, is greater than or equal to it.
81+
its per-invocation scores, across every run, is greater than or equal to it,
82+
unless `multi_run_config` combines the runs another way.
8283

8384
Keys may be written in `snake_case` or `camelCase`, so `match_type` and
8485
`matchType` both parse. That holds for every key in the file, at every level.
@@ -272,11 +273,11 @@ getting that wrong fails loudly. The other is that the key really is
272273
`metric_info`, or `metricInfo`. A `metric` key is not a field, so it is dropped
273274
and the metric silently gets default info instead.
274275

275-
## The other two top-level keys
276+
## The other top-level keys
276277

277-
`EvalConfig` accepts four keys in all. Two of them, `criteria` and
278-
`custom_metrics`, are covered above. The remaining two are
279-
`user_simulator_config` and `live_model_config`.
278+
`EvalConfig` accepts five keys in all. Two of them, `criteria` and
279+
`custom_metrics`, are covered above. The remaining three are
280+
`user_simulator_config`, `live_model_config` and `multi_run_config`.
280281

281282
`user_simulator_config` replaces the recorded user turns with a model that plays
282283
the user against a scenario. Which simulator you get is chosen by a `type`
@@ -298,6 +299,253 @@ audio settings of its own on top of them.
298299
required for Live API models. Its only key is `timeout_seconds`, default `300`,
299300
the time to wait for a model turn to complete.
300301

302+
`multi_run_config` runs each eval case more than once. An agent can answer the
303+
same case differently from one run to the next, so a single run can pass or
304+
fail by chance. `num_runs` says how many times:
305+
306+
```json
307+
{
308+
"criteria": {
309+
"response_match_score": 0.8
310+
},
311+
"multi_run_config": {
312+
"num_runs": 3
313+
}
314+
}
315+
```
316+
317+
With `num_runs` above 1, three things change.
318+
319+
* **Each run starts from a fresh session.** A `session_id` pinned in the eval
320+
case's `session_input` is ignored, with a warning in the log, so every run
321+
starts from the case's initial state instead of from the session an earlier
322+
run left behind.
323+
* **Every run is kept.** Each run produces its own `EvalCaseResult`, and
324+
`adk eval` saves all of them to the eval history. Each result carries the
325+
0-based `repetition_index` of its run, which is always 0 when a case runs
326+
once.
327+
* **The case is judged on all of its runs.** By default each metric's
328+
per-invocation scores from every run are averaged together, and the metric
329+
passes when that mean meets its threshold. A case can therefore pass with a
330+
failing run among its runs, if the others scored high enough to carry the
331+
mean. A run that crashed before it could be scored is left out, and by
332+
default fails the case outright; see [Crashed runs](#crashed-runs).
333+
334+
`adk eval --num_runs N` overrides `num_runs` for one command, and if neither is
335+
set each case runs once. With `--print_detailed_results`, each case's verdict
336+
across its runs is printed first, followed by the results of each of its runs.
337+
`AgentEvaluator` does not read `multi_run_config`; it takes the number of runs
338+
as its own `num_runs` argument, which defaults to `2`.
339+
340+
Unlike a criterion, `multi_run_config` rejects keys it does not recognize, so a
341+
misspelled `num_run` fails when the file is read instead of silently running
342+
each case once. The top-level key itself gets no such check: misspell
343+
`multi_run_config` and the whole entry is dropped.
344+
345+
### How the runs are combined
346+
347+
Three more keys of `multi_run_config` decide how the runs make the verdict.
348+
Each is optional, and they can be combined.
349+
350+
| Key | Judges | Value |
351+
| :--- | :--- | :--- |
352+
| `cross_run_scoring` | Every metric not named in `cross_run_scoring_by_metric`. | A method name, or a rule object. Default `"mean_over_invocations"`. |
353+
| `cross_run_scoring_by_metric` | The metrics it names. Each must be in `criteria`. | A map from metric name to a method name or a rule object. |
354+
| `cross_run_criteria` | The eval case as a whole. | A list of rule objects. |
355+
356+
A **method name** combines scores, and the result is compared with the
357+
metric's threshold in `criteria`:
358+
359+
* `mean_over_invocations`, the default, averages every invocation of every
360+
run. A run with no per-invocation score, as for a metric scored only for
361+
the whole conversation, counts once, with its overall score.
362+
* `mean_of_runs` averages the overall scores of the runs, so each run weighs
363+
the same however many invocations it has.
364+
* `min_over_runs` takes the worst run's score: the lowest, or the highest for
365+
a metric where lower is better. The metric therefore passes only if every
366+
run met its threshold.
367+
368+
As in `AgentEvaluator`, the comparison goes the way the metric's evaluator
369+
judged its scores: if each score it passed is below the threshold and each one
370+
it failed is above, lower is better, so the result passes at or below the
371+
threshold and the output adds `lower is better`; otherwise it passes at or
372+
above the threshold.
373+
374+
A **rule object**, `{"method": ..., "threshold": ...}`, counts the runs that
375+
passed and compares the result with its own `threshold`, from 0 to 1. Each run
376+
is first judged on its own, as a single run would be. The one such method so
377+
far is `pass_rate`: the runs that passed, divided by the runs that passed or
378+
failed. pass@k and pass^k are planned as further methods of this kind.
379+
380+
The case passes when its metrics pass, combined as for a single run, and every
381+
cross-run criterion passes, unless more of its runs crashed than
382+
`max_crashed_runs` allows.
383+
384+
Take one case run three times, scored with `tool_trajectory_avg_score` at a
385+
threshold of 1.0 and `response_match_score` at 0.6:
386+
387+
| Run | `tool_trajectory_avg_score` | `response_match_score` | Whole run |
388+
| :--- | :--- | :--- | :--- |
389+
| 1 | 1.0, passed | 0.875, passed | passed |
390+
| 2 | 1.0, passed | 0.182, failed | failed |
391+
| 3 | 1.0, passed | 0.875, passed | passed |
392+
393+
By default the case passes: `response_match_score` averages 0.644. Each way of
394+
combining the runs can turn that around.
395+
396+
* **By score**, with `{"response_match_score": "min_over_runs"}` in
397+
`cross_run_scoring_by_metric`: the lowest run, 0.182, is below 0.6, so the
398+
metric fails, and with it the case.
399+
* **By how often a metric passed**, with
400+
`{"response_match_score": {"method": "pass_rate", "threshold": 0.9}}` in
401+
`cross_run_scoring_by_metric`: the metric passed in 2 of 3 runs, and 0.67 is
402+
short of 0.9, so the metric fails, and with it the case.
403+
* **By how often the whole case passed**, with
404+
`[{"method": "pass_rate", "threshold": 0.9}]` as `cross_run_criteria`:
405+
2 of 3 runs passed as a whole, so the criterion fails the case although
406+
every metric passes on its mean.
407+
408+
A rule that cannot produce a value, because no run produced a score or no run
409+
passed or failed, is `NOT_EVALUATED`. The output says why, and the rule is
410+
skipped when the verdict is decided, as a metric that was not evaluated is. For
411+
`cross_run_criteria`, a run in which any metric was not evaluated counts as
412+
neither passed nor failed, since the missing result could have been a failure.
413+
414+
The file fails to load when a method name is unknown, when `pass_rate` is given
415+
as a bare name without a threshold, when a method that combines scores is given
416+
as a rule object or in `cross_run_criteria`, when a key of
417+
`cross_run_scoring_by_metric` is not in `criteria`, when a `threshold` is
418+
outside 0 to 1, when `max_crashed_runs` is negative, or when a rule object has
419+
a key it does not recognize.
420+
421+
A rule object judges a case run once as well, so with one configured,
422+
`adk eval --print_detailed_results` prints every case the way it prints a case
423+
run several times: each metric with its method and value, then each cross-run
424+
criterion, then the runs. `AgentEvaluator`, which pytest suites call, and
425+
`adk web` ignore these keys for now.
426+
427+
### Crashed runs
428+
429+
A run crashes when it fails without producing any metric results, which is
430+
what happens when inference raises. Such a run has no verdict, so it is left
431+
out of every count of passing runs, by a metric's rule and by
432+
`cross_run_criteria` alike. `max_crashed_runs`, default `0`, says how many
433+
crashed runs a case tolerates: the case fails if more of its runs crashed than
434+
that. A case also needs at least one run that did not crash to pass, so a case
435+
whose runs all crashed never passes.
436+
437+
```json
438+
{
439+
"criteria": {
440+
"response_match_score": 0.8
441+
},
442+
"multi_run_config": {
443+
"num_runs": 4,
444+
"max_crashed_runs": 1,
445+
"cross_run_criteria": [{"method": "pass_rate", "threshold": 0.75}]
446+
}
447+
}
448+
```
449+
450+
If 3 runs pass and 1 crashes, `pass_rate` is 3 of 3, which is 1.0, and the case
451+
passes. `--print_detailed_results` counts the runs as
452+
`3 of 4 runs passed, 1 crashed`, in the case's header and in the note of every
453+
rule that counts passing runs, and adds
454+
`1 of 4 runs failed without producing any metric results (up to 1 allowed).`
455+
With the default `max_crashed_runs` of 0 the same runs fail the case, and so
456+
would a second crash here, since 2 is more than the 1 allowed. A case run once
457+
without a rule object keeps that run's own verdict, so a crash fails it
458+
whatever `max_crashed_runs` is. `adk eval` does not retry a crashed run.
459+
460+
### Custom methods
461+
462+
When no built-in method fits, write one in Python and declare it under
463+
`custom_methods`. Its key is the name the rules use and the name the output
464+
shows:
465+
466+
```json
467+
{
468+
"criteria": {
469+
"tool_trajectory_avg_score": 1.0,
470+
"response_match_score": 0.6
471+
},
472+
"multi_run_config": {
473+
"num_runs": 5,
474+
"custom_methods": {
475+
"passed_at_least_twice": {
476+
"code_config": {"name": "my_project.eval.methods.passed_at_least_twice"},
477+
"uses_scores": false
478+
},
479+
"median_of_runs": {
480+
"code_config": {"name": "my_project.eval.methods.median_of_runs"},
481+
"uses_scores": true
482+
}
483+
},
484+
"cross_run_scoring": "median_of_runs",
485+
"cross_run_criteria": [
486+
{"method": "passed_at_least_twice", "threshold": 1.0}
487+
]
488+
}
489+
}
490+
```
491+
492+
A method has the signature of a built-in one: it takes one `RunResult` per
493+
run, for one metric or for the whole case, and returns a number, or `None`
494+
when the runs allow no value.
495+
496+
```python
497+
# my_project/eval/methods.py
498+
from collections.abc import Sequence
499+
import statistics
500+
501+
from google.adk.evaluation import RunResult
502+
from google.adk.evaluation.evaluator import EvalStatus
503+
504+
505+
def median_of_runs(runs: Sequence[RunResult]) -> float | None:
506+
scores = [run.score for run in runs if run.score is not None]
507+
return statistics.median(scores) if scores else None
508+
509+
510+
def passed_at_least_twice(runs: Sequence[RunResult]) -> float | None:
511+
passed = sum(1 for run in runs if run.status == EvalStatus.PASSED)
512+
return 1.0 if passed >= 2 else 0.0
513+
```
514+
515+
`uses_scores` is required, and says which kind of method it is:
516+
517+
* **`true`**, a method that combines scores: it reads `score`, the run's
518+
overall score, or `invocation_scores`. It is written as a method name in
519+
`cross_run_scoring` or `cross_run_scoring_by_metric`, and its value is
520+
compared with the metric's threshold in `criteria`. It receives the raw
521+
scores, so a method that picks a run, such as the worst, must handle the
522+
metric's direction itself.
523+
* **`false`**, a method that counts passing runs: it reads `status`. It is
524+
written as a rule object with a threshold of its own, and can also be used
525+
in `cross_run_criteria`, where each `RunResult` is a whole run, with no
526+
score, and a run that crashed is `NOT_EVALUATED`.
527+
528+
The method only computes the number; the threshold and the verdict are applied
529+
as for a built-in method. A method that returns `None` leaves its rule
530+
`NOT_EVALUATED`, with a note, and the rule is skipped when the verdict is
531+
decided. Returning anything else that is not a number, even a `bool`, stops the
532+
eval with a `TypeError` naming the method, and an exception the method raises
533+
is not caught. As with a built-in method, a case run once keeps the run's own
534+
score instead of being re-scored by a method that combines scores.
535+
536+
The file fails to load when a custom method has the name of a built-in one,
537+
when a rule names a method that is neither built in nor declared, or when a
538+
method is written in the form of the other kind. Reading the file does not
539+
import the functions: `adk eval` imports them before it runs any eval case, so
540+
a path that does not import, or names something that is not callable, stops
541+
it at once with an error naming the method. As with the rest of
542+
`multi_run_config`, only `adk eval` reads them.
543+
544+
`code_config` takes no arguments, so a method with parameters is made in Python
545+
and referenced by its full name: with
546+
`pass_at_3 = functools.partial(pass_at_k, k=3)` in `my_project/eval/methods.py`,
547+
the config names `my_project.eval.methods.pass_at_3`.
548+
301549
## How validation actually works
302550

303551
Parsing is deliberately permissive. Every object under `criteria` is validated
@@ -334,7 +582,7 @@ to parse.
334582

335583
* **`EvalConfig` is not re-exported.** Import it as
336584
`from google.adk.evaluation.eval_config import EvalConfig`; the package
337-
`__init__` exports only `AgentEvaluator`.
585+
`__init__` exports only `AgentEvaluator` and `RunResult`.
338586
* **Parsing a config works on a base install; running one does not.**
339587
`eval_config` and `eval_metrics` import with no extras, so you can validate
340588
a file anywhere. The metric registry that turns a metric name into an

‎docs/guides/evaluation/eval_service/index.md‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -243,12 +243,28 @@ runs the agent.
243243
| `labels` | `dict[str, str] \| None` | `None` | Metadata attached for billing breakdown. |
244244
| `use_live` | `bool` | `False` | Use bidirectional streaming. Required for Live API models. |
245245
| `live_timeout_seconds` | `int` | `300` | How long to wait for a model turn in live mode. |
246+
| `num_runs` | `int` | `1` | How many times each eval case is run. Each run is yielded as its own `InferenceResult`. |
246247

247248
`parallelism` is bounded by your model quota, not your CPU. Models enforce
248249
per-minute limits, so raising it on a large eval set is a reliable way to start
249250
collecting rate-limit errors, and those arrive as failed inferences rather than
250251
as an exception you would notice.
251252

253+
`num_runs` above 1 runs every selected case that many times, through the same
254+
`parallelism` pool. Every `InferenceResult` carries a 0-based
255+
`repetition_index` naming the run it came from, 0 when a case runs once, and
256+
`evaluate` copies it onto the matching `EvalCaseResult`, so the runs of one case
257+
stay apart in what is yielded and saved. Results arrive in completion order, so
258+
group the runs of a case by its id and `repetition_index` rather than by
259+
position. With more than one run, each run starts from a fresh session: a
260+
`SessionInput.session_id` pinned by the case is ignored, and a warning naming
261+
the case and the session is logged once per case, so a case that pins its
262+
session to find a pre-loaded artifact will not find it. The service does not
263+
combine the runs into one verdict. That is left to the caller; `adk eval`
264+
judges each case on all of its runs, by default on each metric's
265+
per-invocation scores pooled across the runs, as `multi_run_config` in
266+
[EvalConfig](../eval_config/index.md) describes.
267+
252268
### EvaluateConfig
253269

254270
`EvaluateConfig` travels on an `EvaluateRequest` and governs the scoring phase.
@@ -315,7 +331,11 @@ There are two abstract methods to fill in, both async generators. The usual
315331
reason to write your own is a remote evaluation backend, with inference in your
316332
process and scoring by a hosted service, or the other way round. Keep the
317333
streaming contract when you do, yielding each result as it is ready rather than
318-
collecting a list, because callers rely on that to report progress.
334+
collecting a list, because callers rely on that to report progress. Honor
335+
`InferenceConfig.num_runs` too: yield one `InferenceResult` per run of each
336+
case, labeled with its `repetition_index`, and carry that index onto the
337+
`EvalCaseResult` that `evaluate` yields for it, so callers can tell the runs of
338+
a case apart.
319339

320340
## Limitations
321341

‎docs/guides/evaluation/evaluator/index.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -193,8 +193,13 @@ so `adk eval`, reads the statuses: a case is `FAILED` as soon as any metric
193193
reports `FAILED`, `PASSED` if at least one passed and none failed, and
194194
`NOT_EVALUATED` otherwise. `AgentEvaluator` additionally averages the
195195
per-invocation `score` values across every run and compares that mean against
196-
the configured threshold, which is where its failure messages come from. Setting
197-
both the scores and the statuses, as the example above does, satisfies both.
196+
the configured threshold, which is where its failure messages come from.
197+
`adk eval` averages the same way by default when it runs each case more than
198+
once, through `--num_runs` or `multi_run_config`, and a metric's status then
199+
comes from that mean rather than from the status your metric reported; a rule
200+
in `multi_run_config` that counts passing runs reads the statuses instead.
201+
Setting both the scores and the statuses, as the example above does, satisfies
202+
both.
198203

199204
## Advanced applications
200205

0 commit comments

Comments
 (0)