Skip to content

perf(qa-lab): speed up unified QA suites#95944

Merged
RomneyDa merged 7 commits into
mainfrom
smoke-speed
Jun 23, 2026
Merged

perf(qa-lab): speed up unified QA suites#95944
RomneyDa merged 7 commits into
mainfrom
smoke-speed

Conversation

@RomneyDa

@RomneyDa RomneyDa commented Jun 23, 2026

Copy link
Copy Markdown
Member

What Problem This Solves

The smoke-ci QA profile was taking long enough to block PR 94291. Local profiling showed the slow path was not a single test file: the profile selects 108 scenarios, including 100 flow scenarios, and the unified suite was paying too much serialized lab/provider/gateway time.

This is not smoke-only. The change applies to unified QA suite runs that select both flow scenarios and test-file scenarios, including profile-driven suite runs. Pure flow-only runs keep the existing flow-suite path.

Why This Change Was Made

This keeps all selected scenarios but makes the unified QA launcher run safe work in parallel inside the same CI job:

  • ordinary flow scenarios are split across up to four shared gateway batches
  • scenarios that mutate config/runtime/plugin/memory/image-generation/gateway-state stay isolated
  • scenarios that restart after mutating gateway state now declare execution.suiteIsolation: isolated in YAML instead of depending on a hardcoded call-name list
  • isolated flow scenarios are capped at eight workers and reserve only the workers they can actually use
  • serial unified runs keep multiple isolated flow scenarios in separate flow-suite invocations so they do not share one worker
  • Crabline channel-driver unified runs preserve the existing serial default unless concurrency is explicitly overridden
  • the faster mixed-suite isolated default start stagger is 500ms, while explicit workerStartStaggerMs and OPENCLAW_QA_SUITE_WORKER_START_STAGGER_MS still win
  • Vitest/Playwright/script test-file groups stay serialized in one checkout to avoid local test-runner cache races
  • that serialized test-file group can overlap with flow work when the suite concurrency budget allows it
  • partition scheduling is weighted so --concurrency 1 and other low-resource caps still behave as caps
  • on partition failure, already-started partitions are allowed to settle before the suite rejects, avoiding overlap with retry attempts

Tradeoffs made:

  • This does not add CI shards or new runners; it uses more of the existing job's local concurrency budget.
  • It deliberately leaves known shared-state scenarios isolated, so it is not the theoretical fastest possible schedule.
  • The state-mutation classifier is conservative and structural. New QA scenarios that mutate shared state need to either use explicit execution.suiteIsolation: isolated metadata or an already-classified structural operation.
  • Release-profile runs should benefit where they use the same unified mixed-suite launcher, but release has more live/provider categories and should get its own timing pass before claiming complete release coverage.

User Impact

Unified QA suite profiles should finish materially faster without removing tests and without adding CI shards. Low-resource local runs still have a serial escape hatch through --concurrency 1.

Evidence

Before:

  • node scripts/build-all.mjs qaRuntime: completed successfully; warm runtime build was about 6s locally.
  • Local equivalent unoptimized smoke run: pnpm openclaw qa run --repo-root . --qa-profile smoke-ci ... was interrupted after 486.07s while still incomplete.

After:

  • Full smoke proof before the final concurrency-contract fixes: pnpm openclaw qa run --repo-root . --qa-profile smoke-ci --output-dir .artifacts/qa-e2e/smoke-ci-500-stagger-parallel-nonflow-local-20260622195639 passed 108/108.
  • Raw local command time: 321.27s.
  • Suite timestamps: 2026-06-23T02:56:49.984Z to 2026-06-23T03:02:00.516Z.
  • Focused proof after the final rebase/scheduler/isolation/channel-driver fixes: node scripts/run-vitest.mjs extensions/qa-lab/src/scenario-catalog.test.ts extensions/qa-lab/src/suite-planning.test.ts extensions/qa-lab/src/suite-launch.runtime.test.ts extensions/qa-lab/src/suite.test.ts -- --reporter=verbose passed 89/89.
  • Focused lint: node scripts/run-oxlint.mjs extensions/qa-lab/src/scenario-catalog.ts extensions/qa-lab/src/scenario-catalog.test.ts extensions/qa-lab/src/suite-planning.ts extensions/qa-lab/src/suite-planning.test.ts extensions/qa-lab/src/suite-test-helpers.ts extensions/qa-lab/src/suite-launch.runtime.ts extensions/qa-lab/src/suite-launch.runtime.test.ts passed.
  • git diff --check origin/main...HEAD passed.
  • Autoreview found one post-rebase channel-driver serialization issue; fixed in 10dc5e5 with a regression test. The final autoreview rerun was interrupted by the explicit push request.

Known proof gap: the full 108-scenario smoke run should be rerun on this final commit/CI because the last fixes preserved concurrency contracts, test-file serialization, explicit isolation metadata, and Crabline channel-driver serialization after the full local pass.

@RomneyDa RomneyDa changed the title perf(qa-lab): speed up smoke ci suite perf(qa-lab): speed up unified QA suites Jun 23, 2026
@RomneyDa
RomneyDa merged commit 438f208 into main Jun 23, 2026
87 checks passed
@RomneyDa
RomneyDa deleted the smoke-speed branch June 23, 2026 04:55
LucienShui pushed a commit to LucienShui/openclaw that referenced this pull request Jun 23, 2026
* perf(qa-lab): speed up smoke ci suite

* fix(qa-lab): satisfy suite scheduler lint

* fix(qa-lab): settle unified partitions before retry

* fix(qa-lab): preserve isolated suite safeguards

* refactor(qa-lab): make suite isolation explicit

* fix(qa-lab): preserve channel-driver suite serialization

* fix(qa-lab): narrow flow-only isolation metadata
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 23, 2026
* perf(qa-lab): speed up smoke ci suite

* fix(qa-lab): satisfy suite scheduler lint

* fix(qa-lab): settle unified partitions before retry

* fix(qa-lab): preserve isolated suite safeguards

* refactor(qa-lab): make suite isolation explicit

* fix(qa-lab): preserve channel-driver suite serialization

* fix(qa-lab): narrow flow-only isolation metadata
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant