Skip to content

fix(eval): default judge and user simulator models to LlmAgent.DEFAULT_MODEL - #7733

Open
inamdarmihir wants to merge 1 commit into
google:mainfrom
inamdarmihir:fix/eval-default-judge-model
Open

inamdarmihir wants to merge 1 commit into
google:mainfrom
inamdarmihir:fix/eval-default-judge-model

Conversation

@inamdarmihir

@inamdarmihir inamdarmihir commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem

The eval code defaults to gemini-2.5-flash in three places (JudgeModelOptions.judge_model, LlmBackedUserSimulatorConfig.model, LlmAudioUserSimulatorConfig.model), while LlmAgent.DEFAULT_MODEL is gemini-3.5-flash. A user who runs an eval or a simulated conversation without setting a model gets an id that new projects may not be able to call.

Solution

All three defaults now reference LlmAgent.DEFAULT_MODEL, so the eval defaults follow the agent default and cannot drift apart again. The eval config guide table is updated to match.

The simulators' ThinkingConfig(include_thoughts=True, thinking_budget=10240) is unchanged: the Gemini 3 thinking docs state thinking_budget is still supported for backward compatibility.

Other gemini-2.5-flash defaults (environment_simulation_config.py, the two prompt optimizers) are outside this issue and left alone.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

The new assertions check that JudgeModelOptions().judge_model and LlmBackedUserSimulatorConfig().model equal LlmAgent.DEFAULT_MODEL (the invariant, not a model-name string). The audio simulator default uses the same constant but has no test of its own. tests/unittests/evaluation: 1137 passed.

Manual End-to-End (E2E) Tests:

$ python -c "import google.adk.evaluation, google.adk; \
    from google.adk.evaluation.eval_metrics import JudgeModelOptions as J; \
    from google.adk.evaluation.simulation.llm_backed_user_simulator import LlmBackedUserSimulatorConfig as C; \
    print(J().judge_model, C().model)"
gemini-3.5-flash gemini-3.5-flash

No circular import from evaluation importing LlmAgent. Not run: a live eval against the Gemini API.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Authored by Mihir Inamdar.

…T_MODEL

JudgeModelOptions.judge_model, LlmBackedUserSimulatorConfig.model and
LlmAudioUserSimulatorConfig.model hard-coded "gemini-2.5-flash", which has
drifted from LlmAgent.DEFAULT_MODEL. Reference the agent default directly so
the eval defaults follow it. The simulators' ThinkingConfig(thinking_budget)
is left unchanged: the Gemini 3 docs state thinking_budget is still supported
for backward compatibility.

Fixes google#7649
@google-cla

google-cla Bot commented Oct 26, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@inamdarmihir

Copy link
Copy Markdown
Contributor Author

@googlebot I signed it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluation judge and user simulator default to gemini-2.5-flash, which is no longer available to new users

2 participants