Update QA lab parity gate for GPT-5.5 vs Opus 4.7#74290
Conversation
Greptile SummaryThis PR is a comprehensive rename of the QA lab parity baseline from Confidence Score: 5/5Safe to merge — the change is a consistent model-version rename with no logic gaps and a passing preflight validation. All changed files are mechanical string replacements plus a validated timeout increase. No logic paths were altered, and the PR author ran 527 tests plus an isolated mock preflight. The one observation (waitForOutboundMessage/waitForCondition still at 20 s) is a P2 hardening note, not a blocking bug. No files require special attention.
|
There was a problem hiding this comment.
Pull request overview
Updates OpenClaw’s QA-lab parity gate and associated QA fixtures to consistently compare the current frontier target pair: openai/gpt-5.5 (candidate) vs anthropic/claude-opus-4-7 (baseline), and hardens the mock preflight sentinel by increasing scenario turn timeouts.
Changes:
- Updated CI parity workflows (parity gate, release checks, Convex live-transport gate) to use
openai/gpt-5.5-altandanthropic/claude-opus-4-7, with stable artifact directoriesopenai-candidate/anthropic-baseline. - Updated QA-lab parity/reporting constants, mock provider catalogs, and mock server model defaults to Opus 4.7; refreshed expectations across QA-lab unit/runtime tests.
- Increased
approval-turn-tool-followthroughagent-turn timeout budgets to 60s to reduce cold-start preflight timeouts.
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| test/helpers/auto-reply/trigger-handling-test-harness.ts | Updates mocked model catalog + default config primary model to claude-opus-4-7. |
| qa/scenarios/runtime/approval-turn-tool-followthrough.md | Raises the two agent-turn timeoutMs fallbacks to 60s for preflight robustness. |
| qa/scenarios/models/anthropic-opus-setup-token-smoke.md | Updates Opus smoke scenario text/requiredModel to claude-opus-4-7. |
| qa/scenarios/models/anthropic-opus-api-key-smoke.md | Updates Opus smoke scenario text/requiredModel to claude-opus-4-7. |
| extensions/qa-lab/src/suite.summary-json.test.ts | Updates baseline lane expectations to Opus 4.7. |
| extensions/qa-lab/src/suite-planning.test.ts | Updates requiredModel fixtures to Opus 4.7. |
| extensions/qa-lab/src/qa-gateway-config.test.ts | Updates mock-openai gateway config expectations to include claude-opus-4-7. |
| extensions/qa-lab/src/providers/shared/mock-model-config.ts | Updates mock Anthropic provider catalog default Opus model to claude-opus-4-7. |
| extensions/qa-lab/src/providers/mock-openai/server.ts | Updates mock Anthropic baseline labeling/default model + /v1/models advertisement to Opus 4.7. |
| extensions/qa-lab/src/providers/mock-openai/server.test.ts | Updates mock server tests to assert Opus 4.7 model behavior/advertising. |
| extensions/qa-lab/src/providers/live-frontier/parity.ts | Updates default parity baseline label to anthropic/claude-opus-4-7. |
| extensions/qa-lab/src/providers/live-frontier/character-eval.ts | Updates character-eval model lists/options to Opus 4.7. |
| extensions/qa-lab/src/live-timeout.test.ts | Updates timeout-policy tests to reference Opus 4.7 as the alternate model. |
| extensions/qa-lab/src/cli.runtime.test.ts | Updates CLI runtime tests for parity/judge model defaults to Opus 4.7. |
| extensions/qa-lab/src/character-eval.test.ts | Updates character-eval test fixtures/expectations to Opus 4.7. |
| extensions/qa-lab/src/agentic-parity-report.ts | Updates inline comment examples to Opus 4.7. |
| extensions/qa-lab/src/agentic-parity-report.test.ts | Updates parity-report test fixtures/expected strings to Opus 4.7. |
| .github/workflows/qa-live-transports-convex.yml | Updates mock parity lane model refs and switches artifact dirs to openai-candidate / anthropic-baseline. |
| .github/workflows/parity-gate.yml | Updates parity gate lane models/labels and switches artifact dirs to openai-candidate / anthropic-baseline. |
| .github/workflows/openclaw-release-checks.yml | Updates release parity lanes/models and parity report inputs to the new stable artifact dirs + Opus 4.7 baseline. |
|
Codex review: needs changes before merge. Summary Reproducibility: yes. Source inspection on current main reproduces the stale GPT-5.4/Opus 4.6 workflow labels, artifact paths, QA-lab constants, mock provider defaults, and short preflight waits; I did not run the long mock preflight in this read-only review. Next step before merge Security Review findings
Review detailsBest possible solution: Rebase or replace the PR on current main, drop the deleted standalone parity-gate workflow hunk, update the release guide, preserve the active release/Convex/QA-lab/preflight changes, and land it to close #74262. Do we have a high-confidence way to reproduce the issue? Yes. Source inspection on current main reproduces the stale GPT-5.4/Opus 4.6 workflow labels, artifact paths, QA-lab constants, mock provider defaults, and short preflight waits; I did not run the long mock preflight in this read-only review. Is this the best way to solve the issue? Mostly yes, but not as-is. Updating the owner QA-lab surfaces and aligning the timeout chain is the narrow maintainable fix, but the branch must be rebased away from the deleted parity-gate workflow and paired with release-doc updates before merge. Full review comments:
Overall correctness: patch is incorrect Acceptance criteria:
What I checked:
Likely related people:
Remaining risk / open question:
Codex review notes: model gpt-5.5, reasoning high; reviewed against 66336bf7c846. |
fa3bf0e to
fd03fef
Compare
1149178 to
3c38e8b
Compare
3c38e8b to
f3c8190
Compare
f3c8190 to
ae0838c
Compare
|
gpt 5.4 and 5.5 related @steipete clean update from previous integration by me to 5.5 now. not critical and test only. closing to make room for other PR's, take over and reopen if you want to update to 5.5 test later. note also this test is quite long and can likely be improved @vincentkoc |
|
Accepting @100yenadmin's closure offer above.
A rebase of this PR onto main would conflict on every workflow file and many fixtures; the parity-gate hunks are dead weight against current main. But two payload items from this PR are still missing on main and still load-bearing:
Worth a small follow-up PR with just those two changes (drop everything else from this one — the qa-lab test files already reference |
…approval-turn-tool-followthrough timeouts Carries forward the surface-bump portion of #74290 (closed in favor of this slim follow-up since the parity-gate.yml workflow file the original PR also touched was retired by #74622 'ci: fold parity into QA release validation'). The mock-openai parity lanes that now live in `openclaw-release-checks.yml` and `qa-live-transports-convex.yml` were still pinned to `anthropic/claude-opus-4-6` / `anthropic/claude-sonnet-4-6` for the baseline and `openai/gpt-5.4-alt` for the candidate alt model. That left the parity baseline one model-generation behind the active Opus 4.7 / GPT-5.5 defaults already used elsewhere on main (CHANGELOG.md:803, docs/providers/anthropic.md:108, openclaw-live-and-e2e-checks-reusable.yml:1894). The `approval-turn-tool-followthrough` scenario was using 20s/30s `liveTurnTimeoutMs` fallbacks that timed out on cold mock-gateway parity runs (the deleted `parity-gate.yml` env-var comments described exactly this scenario flake). Bumping all four turn fallbacks to 60s matches what the mock provider's `resolveTurnTimeoutMs` returns for fallbackMs (it returns the fallback unchanged) so cold starts have breathing room before the approval/follow-through chain has to complete. This PR does NOT touch: - The retired `.github/workflows/parity-gate.yml` (deleted on main by #74622) - Internal artifact directory names `gpt54`/`opus46` (cosmetic, out of scope for a slim follow-up) - The Discord QA scenario lane and the release-validation lane that intentionally pin `openai/gpt-5.4` (separate concerns) Refs #74290.
…approval-turn-tool-followthrough timeouts Carries forward the surface-bump portion of #74290 (closed in favor of this slim follow-up since the parity-gate.yml workflow file the original PR also touched was retired by #74622 'ci: fold parity into QA release validation'). The mock-openai parity lanes that now live in `openclaw-release-checks.yml` and `qa-live-transports-convex.yml` were still pinned to `anthropic/claude-opus-4-6` / `anthropic/claude-sonnet-4-6` for the baseline and `openai/gpt-5.4-alt` for the candidate alt model. That left the parity baseline one model-generation behind the active Opus 4.7 / GPT-5.5 defaults already used elsewhere on main (CHANGELOG.md:803, docs/providers/anthropic.md:108, openclaw-live-and-e2e-checks-reusable.yml:1894). The `approval-turn-tool-followthrough` scenario was using 20s/30s `liveTurnTimeoutMs` fallbacks that timed out on cold mock-gateway parity runs (the deleted `parity-gate.yml` env-var comments described exactly this scenario flake). Bumping all four turn fallbacks to 60s matches what the mock provider's `resolveTurnTimeoutMs` returns for fallbackMs (it returns the fallback unchanged) so cold starts have breathing room before the approval/follow-through chain has to complete. This PR does NOT touch: - The retired `.github/workflows/parity-gate.yml` (deleted on main by #74622) - Internal artifact directory names `gpt54`/`opus46` (cosmetic, out of scope for a slim follow-up) - The Discord QA scenario lane and the release-validation lane that intentionally pin `openai/gpt-5.4` (separate concerns) Refs #74290.
Summary
Filed by
100yenadmin.This updates the QA lab parity gates from the mixed GPT-5.4 / Opus 4.6 era naming to the current target comparison:
The PR also hardens the mock preflight sentinel that was timing out on cold startup by giving the full
approval-turn-tool-followthroughturn chain a 60s fallback budget: both agent calls and the paired outbound/wait conditions now align.Fixes #74262.
What Changed
parity-gate, release checks, and the Convex live-transport QA gate:openai/gpt-5.5-altanthropic/claude-opus-4-7openai-candidateandanthropic-baselineclaude-opus-4-7as the active baseline model.approval-turn-tool-followthroughrun-agent and follow-up wait timeouts to 60s so--preflightcan survive a cold mock gateway path.Validation
rg "gpt54|gpt-5\\.4-alt|opus46|opus-4-6|Opus 4\\.6|GPT-5\\.4" .github/workflows extensions/qa-lab qa/scenarios test/helpers/auto-replyreturned no matches on final head.git diff --checkpnpm exec vitest run --config test/vitest/vitest.extension-qa.config.ts extensions/qa-lab/src/providers/mock-openai/server.test.ts extensions/qa-lab/src/qa-gateway-config.test.ts extensions/qa-lab/src/suite-planning.test.ts extensions/qa-lab/src/suite.summary-json.test.ts extensions/qa-lab/src/agentic-parity-report.test.ts extensions/qa-lab/src/character-eval.test.ts extensions/qa-lab/src/cli.runtime.test.ts extensions/qa-lab/src/live-timeout.test.tspnpm exec vitest run --config test/vitest/vitest.extension-qa.config.tsOPENCLAW_BUILD_PRIVATE_QA=1 OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 pnpm buildenv \ HOME=/tmp/openclaw-qa-preflight-home \ OPENCLAW_HOME=/tmp/openclaw-qa-preflight-home \ OPENCLAW_STATE_DIR=/tmp/openclaw-qa-preflight-state \ OPENCLAW_CONFIG_PATH=/tmp/openclaw-qa-preflight-home/openclaw.json \ OPENCLAW_BUILD_PRIVATE_QA=1 \ OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 \ OPENCLAW_QA_SUITE_PROGRESS=1 \ OPENAI_API_KEY= \ ANTHROPIC_API_KEY= \ OPENCLAW_LIVE_OPENAI_KEY= \ OPENCLAW_LIVE_ANTHROPIC_KEY= \ pnpm openclaw qa suite \ --provider-mode mock-openai \ --parity-pack agentic \ --concurrency 1 \ --model openai/gpt-5.5 \ --alt-model openai/gpt-5.5-alt \ --preflightResult before the final main-only rebase:
approval-turn-tool-followthroughpassed 1/1; the final rebase was conflict-free across QA files.pnpm exec vitest run --config test/vitest/vitest.e2e.config.ts src/auto-reply/reply.triggers.trigger-handling.targets-active-session-native-stop.e2e.test.tssrc/auto-replyor the trigger helper conflict-free change.