Skip to content

Commit 138be32

Browse files
committed
build(bench): default the help bench Codex runner to gpt-6-luna
gpt-5.4-mini fails with a ChatGPT login on this host. gpt-6-luna is the latest Codex model; a smoke run of one case completes the call and grades the plan without a runner error.
1 parent 64abfe6 commit 138be32

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

‎scripts/help-conformance-bench.mjs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ const RUN_TIMEOUT_MS = Number(process.env.HELP_BENCH_TIMEOUT_MS ?? 90_000);
2222
// Runner x case pairs run concurrently, capped low: these are paid LLM calls
2323
// and the CLI help subprocess calls behind loadDocs share this same machine.
2424
const CONCURRENCY = Number(process.env.HELP_BENCH_CONCURRENCY ?? 4);
25-
const DEFAULT_RUNNERS = ['codex:gpt-5.4-mini', 'claude:claude-haiku-4-5'];
25+
const DEFAULT_RUNNERS = ['codex:gpt-6-luna', 'claude:claude-haiku-4-5'];
2626
const USAGE = `Usage: node scripts/help-conformance-bench.mjs [options]
2727
2828
Feeds a help slice + task into one non-agentic LLM call per runner x case and

0 commit comments

Comments
 (0)