Skip to content

Commit b0b85b4

Browse files
committed
chore(bench): default the help benchmark's Codex runner to gpt-5.6-luna
gpt-5.4-mini is not available when Codex runs with a ChatGPT login, so every Codex call in a default run failed. gpt-5.6-luna is the small model that the base and head runs for this PR used.
1 parent 55a1ca9 commit b0b85b4

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

‎scripts/help-conformance-bench.mjs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ const RUN_TIMEOUT_MS = Number(process.env.HELP_BENCH_TIMEOUT_MS ?? 90_000);
2222
// Runner x case pairs run concurrently, capped low: these are paid LLM calls
2323
// and the CLI help subprocess calls behind loadDocs share this same machine.
2424
const CONCURRENCY = Number(process.env.HELP_BENCH_CONCURRENCY ?? 4);
25-
const DEFAULT_RUNNERS = ['codex:gpt-5.4-mini', 'claude:claude-haiku-4-5'];
25+
const DEFAULT_RUNNERS = ['codex:gpt-5.6-luna', 'claude:claude-haiku-4-5'];
2626
const USAGE = `Usage: node scripts/help-conformance-bench.mjs [options]
2727
2828
Feeds a help slice + task into one non-agentic LLM call per runner x case and

0 commit comments

Comments
 (0)