Repository navigation
fix(eval): default judge and user simulator models to LlmAgent.DEFAULT_MODEL - #7733
Open
inamdarmihir wants to merge 1 commit into
Open
inamdarmihir wants to merge 1 commit into
inamdarmihir wants to merge 1 commit into
Conversation
…T_MODEL JudgeModelOptions.judge_model, LlmBackedUserSimulatorConfig.model and LlmAudioUserSimulatorConfig.model hard-coded "gemini-2.5-flash", which has drifted from LlmAgent.DEFAULT_MODEL. Reference the agent default directly so the eval defaults follow it. The simulators' ThinkingConfig(thinking_budget) is left unchanged: the Gemini 3 docs state thinking_budget is still supported for backward compatibility. Fixes google#7649
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Contributor
Author
|
@googlebot I signed it! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to Issue or Description of Change
1. Link to an existing issue (if applicable):
Problem
The eval code defaults to
gemini-2.5-flashin three places (JudgeModelOptions.judge_model,LlmBackedUserSimulatorConfig.model,LlmAudioUserSimulatorConfig.model), whileLlmAgent.DEFAULT_MODELisgemini-3.5-flash. A user who runs an eval or a simulated conversation without setting a model gets an id that new projects may not be able to call.Solution
All three defaults now reference
LlmAgent.DEFAULT_MODEL, so the eval defaults follow the agent default and cannot drift apart again. The eval config guide table is updated to match.The simulators'
ThinkingConfig(include_thoughts=True, thinking_budget=10240)is unchanged: the Gemini 3 thinking docs statethinking_budgetis still supported for backward compatibility.Other
gemini-2.5-flashdefaults (environment_simulation_config.py, the two prompt optimizers) are outside this issue and left alone.Testing Plan
Unit Tests:
The new assertions check that
JudgeModelOptions().judge_modelandLlmBackedUserSimulatorConfig().modelequalLlmAgent.DEFAULT_MODEL(the invariant, not a model-name string). The audio simulator default uses the same constant but has no test of its own.tests/unittests/evaluation: 1137 passed.Manual End-to-End (E2E) Tests:
No circular import from
evaluationimportingLlmAgent. Not run: a live eval against the Gemini API.Checklist
Authored by Mihir Inamdar.