Skip to content

Update QA lab parity gate for GPT-5.5 vs Opus 4.7#74290

Closed
100yenadmin wants to merge 2 commits into
openclaw:mainfrom
electricsheephq:qa-lab-gpt55-opus47-preflight
Closed

Update QA lab parity gate for GPT-5.5 vs Opus 4.7#74290
100yenadmin wants to merge 2 commits into
openclaw:mainfrom
electricsheephq:qa-lab-gpt55-opus47-preflight

Conversation

@100yenadmin

@100yenadmin 100yenadmin commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Filed by 100yenadmin.

This updates the QA lab parity gates from the mixed GPT-5.4 / Opus 4.6 era naming to the current target comparison:

openai/gpt-5.5 candidate vs anthropic/claude-opus-4-7 baseline

The PR also hardens the mock preflight sentinel that was timing out on cold startup by giving the full approval-turn-tool-followthrough turn chain a 60s fallback budget: both agent calls and the paired outbound/wait conditions now align.

Fixes #74262.

What Changed

  • Updated parity workflow surfaces in parity-gate, release checks, and the Convex live-transport QA gate:
    • candidate alt: openai/gpt-5.5-alt
    • baseline: anthropic/claude-opus-4-7
    • stable artifact dirs: openai-candidate and anthropic-baseline
  • Updated QA-lab parity/report labels, character eval defaults, summary/report tests, live-timeout tests, and CLI runtime tests for Opus 4.7.
  • Updated the mock Anthropic provider default/model catalog and mock server tests to use claude-opus-4-7 as the active baseline model.
  • Updated Anthropic Opus smoke scenario metadata and the trigger-handling helper catalog/defaults to Opus 4.7.
  • Increased the approval-turn-tool-followthrough run-agent and follow-up wait timeouts to 60s so --preflight can survive a cold mock gateway path.

Validation

  • rg "gpt54|gpt-5\\.4-alt|opus46|opus-4-6|Opus 4\\.6|GPT-5\\.4" .github/workflows extensions/qa-lab qa/scenarios test/helpers/auto-reply returned no matches on final head.
  • git diff --check
  • pnpm exec vitest run --config test/vitest/vitest.extension-qa.config.ts extensions/qa-lab/src/providers/mock-openai/server.test.ts extensions/qa-lab/src/qa-gateway-config.test.ts extensions/qa-lab/src/suite-planning.test.ts extensions/qa-lab/src/suite.summary-json.test.ts extensions/qa-lab/src/agentic-parity-report.test.ts extensions/qa-lab/src/character-eval.test.ts extensions/qa-lab/src/cli.runtime.test.ts extensions/qa-lab/src/live-timeout.test.ts
    • 8 files, 189 tests passed on final head
  • pnpm exec vitest run --config test/vitest/vitest.extension-qa.config.ts
    • 63 files, 527 tests passed before the final main-only rebase; that rebase was conflict-free across QA files
  • OPENCLAW_BUILD_PRIVATE_QA=1 OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 pnpm build
    • passed before the final main-only rebase
  • Isolated mock preflight:
env \
  HOME=/tmp/openclaw-qa-preflight-home \
  OPENCLAW_HOME=/tmp/openclaw-qa-preflight-home \
  OPENCLAW_STATE_DIR=/tmp/openclaw-qa-preflight-state \
  OPENCLAW_CONFIG_PATH=/tmp/openclaw-qa-preflight-home/openclaw.json \
  OPENCLAW_BUILD_PRIVATE_QA=1 \
  OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 \
  OPENCLAW_QA_SUITE_PROGRESS=1 \
  OPENAI_API_KEY= \
  ANTHROPIC_API_KEY= \
  OPENCLAW_LIVE_OPENAI_KEY= \
  OPENCLAW_LIVE_ANTHROPIC_KEY= \
  pnpm openclaw qa suite \
    --provider-mode mock-openai \
    --parity-pack agentic \
    --concurrency 1 \
    --model openai/gpt-5.5 \
    --alt-model openai/gpt-5.5-alt \
    --preflight

Result before the final main-only rebase: approval-turn-tool-followthrough passed 1/1; the final rebase was conflict-free across QA files.

  • pnpm exec vitest run --config test/vitest/vitest.e2e.config.ts src/auto-reply/reply.triggers.trigger-handling.targets-active-session-native-stop.e2e.test.ts
    • 1 file, 20 tests passed before the final rebase; the rebase did not touch src/auto-reply or the trigger helper conflict-free change.

@greptile-apps

greptile-apps Bot commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR is a comprehensive rename of the QA lab parity baseline from anthropic/claude-opus-4-6 / openai/gpt-5.4-alt to anthropic/claude-opus-4-7 / openai/gpt-5.5-alt, touching workflows, source constants, mock server defaults, test fixtures, and scenario metadata consistently. It also increases the two runAgentPrompt fallback budgets in approval-turn-tool-followthrough from 20 s / 30 s to 60 s / 60 s to address cold-gateway preflight timeouts.

Confidence Score: 5/5

Safe to merge — the change is a consistent model-version rename with no logic gaps and a passing preflight validation.

All changed files are mechanical string replacements plus a validated timeout increase. No logic paths were altered, and the PR author ran 527 tests plus an isolated mock preflight. The one observation (waitForOutboundMessage/waitForCondition still at 20 s) is a P2 hardening note, not a blocking bug.

No files require special attention.

Comments Outside Diff (1)

  1. qa/scenarios/runtime/approval-turn-tool-followthrough.md, line 64 (link)

    P2 waitForOutboundMessage / waitForCondition still at 20 s

    The PR bumps both runAgentPrompt fallback budgets to 60 s to survive a cold mock-gateway path, but the two waitForOutboundMessage and waitForCondition steps immediately after each agent turn (lines 64 and 84) still use a 20 s fallback. On a cold gateway the extra latency that was causing the runAgentPrompt timeout could equally push these follow-up wait steps past their budget. Given that the fix works today it may not be urgent, but these are worth aligning to 60 s for consistency.

    Prompt To Fix With AI
    This is a comment left during a code review.
    Path: qa/scenarios/runtime/approval-turn-tool-followthrough.md
    Line: 64
    
    Comment:
    **`waitForOutboundMessage` / `waitForCondition` still at 20 s**
    
    The PR bumps both `runAgentPrompt` fallback budgets to 60 s to survive a cold mock-gateway path, but the two `waitForOutboundMessage` and `waitForCondition` steps immediately after each agent turn (lines 64 and 84) still use a 20 s fallback. On a cold gateway the extra latency that was causing the `runAgentPrompt` timeout could equally push these follow-up wait steps past their budget. Given that the fix works today it may not be urgent, but these are worth aligning to 60 s for consistency.
    
    How can I resolve this? If you propose a fix, please make it concise.

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Prompt To Fix All With AI
This is a comment left during a code review.
Path: qa/scenarios/runtime/approval-turn-tool-followthrough.md
Line: 64

Comment:
**`waitForOutboundMessage` / `waitForCondition` still at 20 s**

The PR bumps both `runAgentPrompt` fallback budgets to 60 s to survive a cold mock-gateway path, but the two `waitForOutboundMessage` and `waitForCondition` steps immediately after each agent turn (lines 64 and 84) still use a 20 s fallback. On a cold gateway the extra latency that was causing the `runAgentPrompt` timeout could equally push these follow-up wait steps past their budget. Given that the fix works today it may not be urgent, but these are worth aligning to 60 s for consistency.

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "qa-lab: update parity gate for opus 4.7" | Re-trigger Greptile

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates OpenClaw’s QA-lab parity gate and associated QA fixtures to consistently compare the current frontier target pair: openai/gpt-5.5 (candidate) vs anthropic/claude-opus-4-7 (baseline), and hardens the mock preflight sentinel by increasing scenario turn timeouts.

Changes:

  • Updated CI parity workflows (parity gate, release checks, Convex live-transport gate) to use openai/gpt-5.5-alt and anthropic/claude-opus-4-7, with stable artifact directories openai-candidate / anthropic-baseline.
  • Updated QA-lab parity/reporting constants, mock provider catalogs, and mock server model defaults to Opus 4.7; refreshed expectations across QA-lab unit/runtime tests.
  • Increased approval-turn-tool-followthrough agent-turn timeout budgets to 60s to reduce cold-start preflight timeouts.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated no comments.

Show a summary per file
File Description
test/helpers/auto-reply/trigger-handling-test-harness.ts Updates mocked model catalog + default config primary model to claude-opus-4-7.
qa/scenarios/runtime/approval-turn-tool-followthrough.md Raises the two agent-turn timeoutMs fallbacks to 60s for preflight robustness.
qa/scenarios/models/anthropic-opus-setup-token-smoke.md Updates Opus smoke scenario text/requiredModel to claude-opus-4-7.
qa/scenarios/models/anthropic-opus-api-key-smoke.md Updates Opus smoke scenario text/requiredModel to claude-opus-4-7.
extensions/qa-lab/src/suite.summary-json.test.ts Updates baseline lane expectations to Opus 4.7.
extensions/qa-lab/src/suite-planning.test.ts Updates requiredModel fixtures to Opus 4.7.
extensions/qa-lab/src/qa-gateway-config.test.ts Updates mock-openai gateway config expectations to include claude-opus-4-7.
extensions/qa-lab/src/providers/shared/mock-model-config.ts Updates mock Anthropic provider catalog default Opus model to claude-opus-4-7.
extensions/qa-lab/src/providers/mock-openai/server.ts Updates mock Anthropic baseline labeling/default model + /v1/models advertisement to Opus 4.7.
extensions/qa-lab/src/providers/mock-openai/server.test.ts Updates mock server tests to assert Opus 4.7 model behavior/advertising.
extensions/qa-lab/src/providers/live-frontier/parity.ts Updates default parity baseline label to anthropic/claude-opus-4-7.
extensions/qa-lab/src/providers/live-frontier/character-eval.ts Updates character-eval model lists/options to Opus 4.7.
extensions/qa-lab/src/live-timeout.test.ts Updates timeout-policy tests to reference Opus 4.7 as the alternate model.
extensions/qa-lab/src/cli.runtime.test.ts Updates CLI runtime tests for parity/judge model defaults to Opus 4.7.
extensions/qa-lab/src/character-eval.test.ts Updates character-eval test fixtures/expectations to Opus 4.7.
extensions/qa-lab/src/agentic-parity-report.ts Updates inline comment examples to Opus 4.7.
extensions/qa-lab/src/agentic-parity-report.test.ts Updates parity-report test fixtures/expected strings to Opus 4.7.
.github/workflows/qa-live-transports-convex.yml Updates mock parity lane model refs and switches artifact dirs to openai-candidate / anthropic-baseline.
.github/workflows/parity-gate.yml Updates parity gate lane models/labels and switches artifact dirs to openai-candidate / anthropic-baseline.
.github/workflows/openclaw-release-checks.yml Updates release parity lanes/models and parity report inputs to the new stable artifact dirs + Opus 4.7 baseline.

@clawsweeper

clawsweeper Bot commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs changes before merge.

Summary
The PR updates QA-lab parity workflows, mock provider defaults/tests, scenario metadata, trigger fixtures, and approval-followthrough timeouts from GPT-5.4/Opus 4.6-era values to GPT-5.5/Opus 4.7 with generic artifact dirs.

Reproducibility: yes. Source inspection on current main reproduces the stale GPT-5.4/Opus 4.6 workflow labels, artifact paths, QA-lab constants, mock provider defaults, and short preflight waits; I did not run the long mock preflight in this read-only review.

Next step before merge
The remaining work is narrow and mechanical enough for a repair pass: rebase or replace the branch, remove the deleted workflow hunk, update release docs, and rerun focused QA validation.

Security
Cleared: Security review cleared: the diff changes QA workflow arguments, model labels, artifact paths, tests, scenario metadata, and timeout budgets without broadening permissions, secrets access, dependency resolution, or third-party execution.

Review findings

  • [P2] Resolve the deleted parity-gate workflow conflict — .github/workflows/parity-gate.yml:24
  • [P3] Update the release QA guide for Opus 4.7 — .github/workflows/openclaw-release-checks.yml:611
Review details

Best possible solution:

Rebase or replace the PR on current main, drop the deleted standalone parity-gate workflow hunk, update the release guide, preserve the active release/Convex/QA-lab/preflight changes, and land it to close #74262.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection on current main reproduces the stale GPT-5.4/Opus 4.6 workflow labels, artifact paths, QA-lab constants, mock provider defaults, and short preflight waits; I did not run the long mock preflight in this read-only review.

Is this the best way to solve the issue?

Mostly yes, but not as-is. Updating the owner QA-lab surfaces and aligning the timeout chain is the narrow maintainable fix, but the branch must be rebased away from the deleted parity-gate workflow and paired with release-doc updates before merge.

Full review comments:

  • [P2] Resolve the deleted parity-gate workflow conflict — .github/workflows/parity-gate.yml:24
    Current main no longer has .github/workflows/parity-gate.yml, but this PR still modifies that workflow. Rebase the branch and keep the parity changes in the active release and Convex QA workflows instead of reintroducing the deleted standalone gate.
    Confidence: 0.9
  • [P3] Update the release QA guide for Opus 4.7 — .github/workflows/openclaw-release-checks.yml:611
    The PR changes the release QA parity baseline and artifact paths, but docs/reference/RELEASING.md still says the release QA Lab mock parity lane compares against Opus 4.6. Update the guide so release operators see the same target comparison the workflow will run.
    Confidence: 0.86

Overall correctness: patch is incorrect
Overall confidence: 0.88

Acceptance criteria:

  • git diff --check
  • pnpm docs:list
  • rg "gpt54|gpt-5\.4-alt|opus46|opus-4-6|Opus 4\.6|GPT-5\.4" .github/workflows extensions/qa-lab qa/scenarios test/helpers/auto-reply docs/reference/RELEASING.md
  • pnpm exec vitest run --config test/vitest/vitest.extension-qa.config.ts extensions/qa-lab/src/providers/mock-openai/server.test.ts extensions/qa-lab/src/qa-gateway-config.test.ts extensions/qa-lab/src/suite-planning.test.ts extensions/qa-lab/src/suite.summary-json.test.ts extensions/qa-lab/src/agentic-parity-report.test.ts extensions/qa-lab/src/character-eval.test.ts extensions/qa-lab/src/cli.runtime.test.ts extensions/qa-lab/src/live-timeout.test.ts
  • OPENCLAW_BUILD_PRIVATE_QA=1 OPENCLAW_ENABLE_PRIVATE_QA_CLI=1 pnpm build

What I checked:

Likely related people:

  • vincentkoc: Sampled current-main blame and shortlog for the QA-lab parity and release-lane surfaces point to Vincent Koc as a recent maintainer, and the PR author explicitly tagged them for possible follow-up on the long QA test. (role: recent maintainer / likely follow-up owner; confidence: medium; commits: 63ebe372e8fb; files: .github/workflows/openclaw-release-checks.yml, .github/workflows/qa-live-transports-convex.yml, extensions/qa-lab/src/providers/live-frontier/parity.ts)
  • Onur: Local workflow history shows Onur authored recent release-check workflow split/stabilization commits that include the active release QA parity workflow touched by this PR. (role: release workflow history owner; confidence: medium; commits: cdcdb4bb93c0, 900e291f31fd, 51606e98894c; files: .github/workflows/openclaw-release-checks.yml)
  • steipete: The PR author tagged Peter Steinberger as a take-over/review candidate; local shortlog sampling also shows prior touches in the relevant broad QA/workflow area, but the concrete file-history trail is weaker than for the workflow authors. (role: adjacent maintainer reviewer; confidence: low; files: .github/workflows/openclaw-release-checks.yml, extensions/qa-lab)

Remaining risk / open question:

  • The branch cannot be treated as ready while it modifies .github/workflows/parity-gate.yml, which current main no longer contains.
  • Release operators would still see Opus 4.6 in docs if the workflow update lands without the matching release guide edit.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 66336bf7c846.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

Comment thread .github/workflows/parity-gate.yml Outdated
Comment thread .github/workflows/qa-live-transports-convex.yml Outdated
Comment thread .github/workflows/openclaw-release-checks.yml
@100yenadmin
100yenadmin force-pushed the qa-lab-gpt55-opus47-preflight branch 3 times, most recently from 1149178 to 3c38e8b Compare April 29, 2026 11:36
@openclaw-barnacle openclaw-barnacle Bot added the commands Command implementations label Apr 29, 2026
@100yenadmin
100yenadmin force-pushed the qa-lab-gpt55-opus47-preflight branch from 3c38e8b to f3c8190 Compare April 29, 2026 11:43
@openclaw-barnacle openclaw-barnacle Bot removed the commands Command implementations label Apr 29, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.

@100yenadmin

Copy link
Copy Markdown
Contributor Author

gpt 5.4 and 5.5 related @steipete clean update from previous integration by me to 5.5 now. not critical and test only. closing to make room for other PR's, take over and reopen if you want to update to 5.5 test later. note also this test is quite long and can likely be improved @vincentkoc

@100yenadmin

Copy link
Copy Markdown
Contributor Author

Accepting @100yenadmin's closure offer above.

.github/workflows/parity-gate.yml was deliberately retired on main by #74622 ("ci: fold parity into QA release validation", b9eb31b). PR-time parity was already removed earlier by 666ab0a "ci: stop parity gate on pull requests". The same qa suite --parity-pack agentic candidate-vs-baseline now lives in two places:

  • .github/workflows/openclaw-release-checks.yml:652-789qa_lab_parity_lane_release_checks + qa_lab_parity_report_release_checks (manual / release path).
  • .github/workflows/qa-live-transports-convex.yml:151-217run_mock_parity nightly cron 41 4 * * *.

A rebase of this PR onto main would conflict on every workflow file and many fixtures; the parity-gate hunks are dead weight against current main.

But two payload items from this PR are still missing on main and still load-bearing:

  1. Stale baseline: openclaw-release-checks.yml:711,782 and qa-live-transports-convex.yml:199,210 still compare against anthropic/claude-opus-4-6, while Opus 4.7 is the active default elsewhere on main (docs/providers/anthropic.md:108,262, openclaw-live-and-e2e-checks-reusable.yml:1894, CHANGELOG.md:803). The parity baseline is one model-generation behind.
  2. Cold-startup flake: qa/scenarios/runtime/approval-turn-tool-followthrough.md:57,64,75,84 still uses 20s/20s/30s/20s timeouts. The deleted workflow's own env-var comments described this scenario flaking on the cold mock-gateway path.

Worth a small follow-up PR with just those two changes (drop everything else from this one — the qa-lab test files already reference gpt-5.5-alt on main).

@100yenadmin 100yenadmin closed this May 8, 2026
steipete pushed a commit that referenced this pull request May 9, 2026
…approval-turn-tool-followthrough timeouts

Carries forward the surface-bump portion of #74290 (closed in favor of
this slim follow-up since the parity-gate.yml workflow file the original
PR also touched was retired by #74622 'ci: fold parity into QA release
validation').

The mock-openai parity lanes that now live in
`openclaw-release-checks.yml` and `qa-live-transports-convex.yml`
were still pinned to `anthropic/claude-opus-4-6` /
`anthropic/claude-sonnet-4-6` for the baseline and
`openai/gpt-5.4-alt` for the candidate alt model. That left the parity
baseline one model-generation behind the active Opus 4.7 / GPT-5.5
defaults already used elsewhere on main (CHANGELOG.md:803,
docs/providers/anthropic.md:108, openclaw-live-and-e2e-checks-reusable.yml:1894).

The `approval-turn-tool-followthrough` scenario was using 20s/30s
`liveTurnTimeoutMs` fallbacks that timed out on cold mock-gateway
parity runs (the deleted `parity-gate.yml` env-var comments described
exactly this scenario flake). Bumping all four turn fallbacks to 60s
matches what the mock provider's `resolveTurnTimeoutMs` returns for
fallbackMs (it returns the fallback unchanged) so cold starts have
breathing room before the approval/follow-through chain has to
complete.

This PR does NOT touch:
- The retired `.github/workflows/parity-gate.yml` (deleted on main
  by #74622)
- Internal artifact directory names `gpt54`/`opus46` (cosmetic, out
  of scope for a slim follow-up)
- The Discord QA scenario lane and the release-validation lane that
  intentionally pin `openai/gpt-5.4` (separate concerns)

Refs #74290.
steipete pushed a commit that referenced this pull request May 9, 2026
…approval-turn-tool-followthrough timeouts

Carries forward the surface-bump portion of #74290 (closed in favor of
this slim follow-up since the parity-gate.yml workflow file the original
PR also touched was retired by #74622 'ci: fold parity into QA release
validation').

The mock-openai parity lanes that now live in
`openclaw-release-checks.yml` and `qa-live-transports-convex.yml`
were still pinned to `anthropic/claude-opus-4-6` /
`anthropic/claude-sonnet-4-6` for the baseline and
`openai/gpt-5.4-alt` for the candidate alt model. That left the parity
baseline one model-generation behind the active Opus 4.7 / GPT-5.5
defaults already used elsewhere on main (CHANGELOG.md:803,
docs/providers/anthropic.md:108, openclaw-live-and-e2e-checks-reusable.yml:1894).

The `approval-turn-tool-followthrough` scenario was using 20s/30s
`liveTurnTimeoutMs` fallbacks that timed out on cold mock-gateway
parity runs (the deleted `parity-gate.yml` env-var comments described
exactly this scenario flake). Bumping all four turn fallbacks to 60s
matches what the mock provider's `resolveTurnTimeoutMs` returns for
fallbackMs (it returns the fallback unchanged) so cold starts have
breathing room before the approval/follow-through chain has to
complete.

This PR does NOT touch:
- The retired `.github/workflows/parity-gate.yml` (deleted on main
  by #74622)
- Internal artifact directory names `gpt54`/`opus46` (cosmetic, out
  of scope for a slim follow-up)
- The Discord QA scenario lane and the release-validation lane that
  intentionally pin `openai/gpt-5.4` (separate concerns)

Refs #74290.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update QA lab parity gate for GPT-5.5 vs Opus 4.7 and harden preflight

2 participants