Skip to content

ci: shard automatic QA smoke by channel#101164

Closed
RomneyDa wants to merge 2 commits into
mainfrom
qa-smoke-faster
Closed

ci: shard automatic QA smoke by channel#101164
RomneyDa wants to merge 2 commits into
mainfrom
qa-smoke-faster

Conversation

@RomneyDa

@RomneyDa RomneyDa commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What Problem This Solves

Automatically triggered QA Smoke CI on pull requests and main pushes regressed from roughly 7 minutes to 10–14 minutes after isolated channel-driver workers became serial. The current smoke-ci profile also now selects Telegram, Matrix, Slack, and WhatsApp scenarios in one suite, so automatic CI aborts after packaging with Selected QA scenarios require multiple channels ...; split the run by channel.

Failing automatic run: https://github.com/openclaw/openclaw/actions/runs/28818875321/job/85465314865

Why This Change Was Made

  • Plan the 92 current smoke scenarios from the canonical taxonomy and provider-lane rules.
  • Group scenarios by their effective Crabline channel; unbound scenarios use Crabline's default Telegram channel.
  • Split the 87-scenario Telegram group into two balanced shards, with one shard each for Matrix, Slack, and WhatsApp.
  • Run the five shards in parallel on the existing automatic Blacksmith 16-vCPU runner policy; manual and fork fallbacks remain GitHub-hosted.
  • Preserve the required QA Smoke CI status with a small GitHub-hosted aggregate job.
  • Keep fanout bounded at five jobs. This replaces one Blacksmith QA registration with five (+4 per full CI run, about 3.7% against a representative 107-job full run) rather than restoring the old category-per-job fanout.

User Impact

Automatic QA Smoke no longer fails from mixed channel selection. Its critical path is reduced from one 92-scenario serial-isolation workload to two balanced 43/44-scenario Telegram workloads running alongside the three small channel shards. The actual end-to-end automatic timing will be visible on this PR's refreshed CI run.

Evidence

  • Blacksmith Testbox tbx_01kwwgbf85anatvnc14b5frs7b:
    • corepack pnpm test extensions/qa-lab/src/ci-smoke-plan.test.ts test/scripts/ci-workflow-guards.test.ts — 35 tests passed.
    • node scripts/check-workflows.mjs — passed (zizmor and workflow interpolation checks).
    • Real Matrix shard with package creation and three QA scenarios — passed in 2m48s and produced QA evidence.
  • Local actionlint .github/workflows/ci.yml — passed.
  • git diff --check — passed.
  • Fresh autoreview — clean after fixing its preflight dependency finding.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation size: XS maintainer Maintainer-authored PR labels Jul 6, 2026
@RomneyDa RomneyDa changed the title ci: speed up manual QA smoke checks ci: shard automatic QA smoke by channel Jul 6, 2026
@steipete

steipete commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Superseded by #101173, now landed as b773f79.

That fix keeps automatic Crabline profiles on the single default channel at the profile-selection owner boundary while preserving explicit non-default channel requests. Exact-head CI run 28820232195 passed, including the required QA Smoke CI job: https://github.com/openclaw/openclaw/actions/runs/28820232195/job/85469901565

With current main, the mixed-channel planning failure is resolved without additional workflow fanout. Closing this alternative to keep one canonical path. Thank you for driving the diagnosis and follow-up fix, @RomneyDa.

@steipete steipete closed this Jul 6, 2026
@RomneyDa RomneyDa self-assigned this Jul 6, 2026
@steipete

steipete commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Correction: #101173 was immediately reverted by #101184, so the mixed-channel automatic smoke failure is present on current main again.

Reopening this sharding path. The refreshed branch also carries fixes found by its first real exact-head run: register the workflow-only planner with the dead-code gate, keep the unsupported Crabline Slack restart scenario on the live driver, and isolate the silence-sensitive channel baseline from concurrent outbound traffic.

@RomneyDa

RomneyDa commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

can't reopen this haha but opening a new one

@RomneyDa RomneyDa mentioned this pull request Jul 6, 2026
Merged
@steipete steipete mentioned this pull request Jul 6, 2026
Merged
@RomneyDa RomneyDa mentioned this pull request Jul 6, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment