Repository navigation
fix(terminal_bench): skip max_turns for Codex agent (Harbor 0.24.0 compatibility) - #1846
Merged
Merged
Conversation
Harbor 0.24.0's Codex agent schema rejects the max_turns parameter, causing Terminal-Bench evaluation to fail at agent initialization. Skip passing max_turns when agent_name is 'codex'; all other agents retain existing behavior. Closes modelscope#1836
Yunnglin
added a commit
that referenced
this pull request
Oct 13, 2026
Trailing formatting drift from #1846: ruff-format 0.16.4 (pinned by pre-commit) requires the single-line form; local older ruff accepted the wrapped form, so CI --all-files flagged it.
Yunnglin
added a commit
that referenced
this pull request
Oct 13, 2026
…1.1) (#1848) * fix(ai2d): use per-question images per official abcLabel semantics (v1.1) The lmms-lab/ai2d dataset stores one image version per diagram, and 571/814 diagrams kept the letter-masked variant. Since abcLabel is a per-question flag in the official release, 725 questions whose answers are diagram text were shown masked images where that text is erased (#1847). Ship a corrected dataset at modelscope/evalscope/ai2d built from the official ai2d-all.zip annotations (identical selection logic to VLMEvalKit AI2D_TEST, source TSV MD5-verified): letter-referencing questions get the masked image, all others the original. Bump evaluation_version to v1.1 and encode images with the sniffed MIME type (the new dataset is JPEG, previously hard-coded as PNG). * fix(docs): switch benchmark description translation model to qwen-max * style(terminal_bench): collapse agent_kwargs ternary per ruff 0.16.4 Trailing formatting drift from #1846: ruff-format 0.16.4 (pinned by pre-commit) requires the single-line form; local older ruff accepted the wrapped form, so CI --all-files flagged it. * docs(ai2d): drop version-history note from benchmark description Per review feedback: keep the evaluation notes free of v1.0-vs-v1.1 migration detail; the semantics are tracked by evaluation_version. * fix(docs): protect fenced code blocks in README translation Copilot review: translate_readme sent code blocks to the model with no protection, so a translation could persist altered JSON samples and prompt templates (observed with the ai2d zh metadata). Protect blocks with placeholders and fail closed if the model alters one; realign the ai2d zh docs with the metadata. Also drop the stale JPEG/PNG-mix claim in the ai2d comment - the corrected dataset is JPEG-only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When running Terminal-Bench 4.0 with
agent_name='codex', evaluation fails at agent initialization with:Harbor 0.24.0 (within the supported range
harbor>=0.14.0,<1.0.0) does not acceptmax_turnsfor the Codex agent. EvalScope unconditionally passesmax_turnsto all agents viaAgentConfig.kwargs, even when the value isNone(Terminal-Bench 4.0 default).Solution
Skip passing
max_turnswhenagent_name == 'codex', while keeping the existing behavior for all other agents (terminus-2, claude-code, qwen-coder, openhands, opencode, mini-swe-agent, oracle).Changes
evalscope/benchmarks/terminal_bench/terminal_bench_adapter.py: In_on_inference(), conditionally constructagent_kwargs— empty dict for Codex,{'max_turns': self.max_turns}otherwise.Testing
Environment
Verification
agent_name == 'codex'path; all other agent configurations are byte-for-byte unchanged.Pre-existing Failures
Notes for Reviewer
COMMON_EXTRA_PARAMS(max_turns default 200); v4 usesV4_EXTRA_PARAMS(max_turns default None). Both paths share the same_on_inference()method, so this fix covers all versions.Closes #1836