Skip to content

ci: run QA smoke profile in CI#94291

Merged
RomneyDa merged 23 commits into
openclaw:mainfrom
Solvely-Colin:codex/qa-lab-smoke-profile-action
Jun 24, 2026
Merged

ci: run QA smoke profile in CI#94291
RomneyDa merged 23 commits into
openclaw:mainfrom
Solvely-Colin:codex/qa-lab-smoke-profile-action

Conversation

@Solvely-Colin

@Solvely-Colin Solvely-Colin commented Jun 17, 2026 •
edited by RomneyDa
Loading

Copy link
Copy Markdown
Contributor

What Problem This Solves

PR #95880 generalized QA profile evidence generation, but this smoke proof should run through normal CI planning rather than a separate manual wrapper or a category fanout. This PR folds the mock-backed smoke-ci QA profile into the existing checks-fast-bundled-protocol CI shard.

Why This Change Was Made

smoke-ci uses mock-provider mode, so it does not need OPENAI_API_KEY. The CI workflow now builds the private QA runtime and runs pnpm openclaw qa run --qa-profile smoke-ci --concurrency 8 from the existing bundled-protocol fast-core task, then uploads .artifacts/qa-e2e/smoke-ci-profile/ from that same job.

The workflow does not re-list smoke category IDs from taxonomy.yaml, does not add qa_category matrix entries, and does not create checks-fast-qa-smoke-profile-* checks. The guard test asserts those constraints so category taxonomy stays single-sourced in taxonomy.yaml.

The explicit --concurrency 8 is intentional: smoke-ci uses the Crabline-backed channel driver, and #95944 preserves Crabline unified-suite serial execution unless concurrency is explicitly overridden. The previous no-fanout CI run proved the shape but took 12m51s because it logged run start: scenarios=52 concurrency=1 and then ran the isolated flow partitions one at a time.

This branch is rebased over #96320, so the smoke profile cleanup lives separately and this PR remains scoped to the CI integration.

User Impact

No product runtime behavior changes. Full Node CI now produces QA smoke evidence from an existing fast-core shard without registering one runner per QA category. Scenario failures preserve the QA exit code while the always() artifact upload still keeps evidence for debugging.

Evidence

Current PR head: e520229d2e7c8e095c6ec5aee16b39e432a780aa.

Local checks on the rebased branch:

  • git diff --check
  • node scripts/run-vitest.mjs test/scripts/ci-workflow-guards.test.ts
  • go run github.com/rhysd/actionlint/cmd/[email protected] .github/workflows/ci.yml
  • smoke profile selection check: smokeCategoryCount=30, smokeScenarioCount=80, selectedSlowScenarios=[]

Prior local and CI smoke timings/proof:

  • Pre-cleanup full unsplit local smoke-ci: 1631s, selected 34 categories / 86 scenarios, produced evidence but failed 59 passed / 27 failed before the perf(qa-lab): speed up unified QA suites #95944 optimizer work and test(qa): clean up smoke taxonomy profile #96320 taxonomy cleanup were folded in.
  • Rejected category fanout CI run: cancelled run 28080234325 after it created one check per QA category.
  • No-fanout serial CI run: checks-fast-bundled-protocol in run 28080594830 succeeded, uploaded qa-smoke-profile-28080594830-1, and took 12m51s because Crabline defaulted to concurrency=1.
  • Narrow category proof gateway-runtime.websocket-connection, no --allow-failures, including qaRuntime build: 14s and passed. This proves native execution.kind=vitest QA scenarios work through this profile path.

Linked Context

Related #94276
Related #94283
Related #94306
Related #95880
Related #95944
Related #95971
Related #96320

Risk Checklist

  • User-visible behavior: no product runtime behavior changed.
  • Config/environment/migration behavior: no runtime config or migration behavior changed.
  • Security/auth/secrets/tool execution: CI smoke does not pass live provider secrets and relies on mock-provider mode.
  • Highest-risk area: adding full QA smoke work to an existing Node CI shard.
  • Mitigation: no new runner fanout, explicit Crabline concurrency to use perf(qa-lab): speed up unified QA suites #95944's in-job scheduler, artifact upload scoped to bundled-protocol, required-check failure on scenario failure, and a guard test tying the workflow to the smoke-ci taxonomy profile without duplicating category IDs.

@openclaw-barnacle openclaw-barnacle Bot added scripts Repository scripts extensions: qa-lab size: XL triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 17, 2026
@clawsweeper

clawsweeper Bot commented Jun 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed June 24, 2026, 12:43 PM ET / 16:43 UTC.

Summary
The branch adds a separate QA Smoke CI fast-core matrix task that runs the smoke-ci QA profile with evidence upload, updates CI docs, and extends the workflow guard test.

PR surface: Tests +35, Config +26. Total +61 across 2 files.

Reproducibility: not applicable. as a bug reproduction; the relevant proof path is the latest exact-head GitHub Actions QA Smoke CI run and artifact output.

Review metrics: 1 noteworthy metric.

  • CI check topology: 1 fast-core matrix task added. The PR adds a distinct CI check, so maintainers should verify branch-protection expectations and the new lane’s runtime before merge.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🐚 platinum hermit
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Wait for a green exact-head QA Smoke CI run with the qa-smoke-profile-* artifact ID.
  • Update the PR body’s Evidence section from the older head SHA to the latest successful run.

Proof guidance:

  • [P1] Needs stronger real behavior proof before merge: Latest-head proof is not sufficient yet because QA Smoke CI is still in progress and no qa-smoke-profile-* artifact is available for the final head; screenshots or videos are not needed here, but CI logs/artifacts should be redacted if they include private details. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

  • [P1] The new exact-head QA Smoke CI check is still running, so there is not yet green real behavior proof for the final topology.
  • [P1] The PR body’s Evidence section still names an older head SHA, so the durable contributor-facing proof should be refreshed after the latest run finishes.
  • [P1] Adding a new named CI matrix check may require maintainer awareness of required-check and dashboard expectations before merge.

Maintainer options:

  1. Wait For QA Smoke CI (recommended)
    Let the latest exact-head QA Smoke CI job finish and use its artifact as the merge proof.
  2. Repair Or Narrow Smoke
    If QA Smoke CI fails or hangs, fix the failing scenario or narrow the profile before keeping the fail-closed check.
  3. Accept New Check Deliberately
    Maintainers can accept the new check topology once they are comfortable with the required-check and CI-dashboard impact.

Next step before merge

  • [P1] The remaining action is latest-head CI/proof plus protected-label maintainer handling, not a narrow automated repair task.

Security
Cleared: No concrete security or supply-chain concern found; the workflow keeps read-only token behavior, adds no secrets, and uses a pinned upload-artifact action.

Review details

Best possible solution:

Keep the split QA Smoke CI topology only after the latest exact-head smoke check finishes green and uploads the expected qa-smoke-profile-* artifact.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a bug reproduction; the relevant proof path is the latest exact-head GitHub Actions QA Smoke CI run and artifact output.

Is this the best way to solve the issue?

Yes conditionally: splitting smoke into its own matrix task is the safer shape compared with folding it into checks-fast-bundled-protocol, but it still needs green exact-head proof.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against c5f10b5f7c00.

Label changes

Label justifications:

  • P2: The PR changes CI validation topology and QA smoke coverage without direct product runtime impact.
  • merge-risk: 🚨 automation: The diff adds a new CI matrix check and artifact path whose reliability is not settled until exact-head smoke proof completes.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: Latest-head proof is not sufficient yet because QA Smoke CI is still in progress and no qa-smoke-profile-* artifact is available for the final head; screenshots or videos are not needed here, but CI logs/artifacts should be redacted if they include private details. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Tests +35, Config +26. Total +61 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 0 0 0 0
Tests 1 35 0 +35
Docs 0 0 0 0
Config 1 27 1 +26
Generated 0 0 0 0
Other 0 0 0 0
Total 2 62 1 +61

What I checked:

  • Repository policy read: Root AGENTS.md was read fully and applied; scoped test, scripts, and docs policies were also read for the touched paths. (AGENTS.md:1)
  • Latest PR diff: The latest head adds a QA Smoke CI matrix task, a qa-smoke-ci) case, an upload step gated to that task, CI docs text, and guard-test assertions. (.github/workflows/ci.yml:284, 0455bddb70a0)
  • Current main behavior: Current main has checks-fast-bundled-protocol running only pnpm test:bundled and pnpm protocol:check; no QA smoke CI task is present there. (.github/workflows/ci.yml:921, c5f10b5f7c00)
  • Latest exact-head CI state: For head 0455bdd, checks-fast-bundled-protocol, check-guards, and actionlint are green, but QA Smoke CI remains in progress. (0455bddb70a0)
  • Smoke prerequisite history: Merged PR 96340 fixed and sped up the smoke profile, reporting a 78/78 full smoke pass in 2:51.97; this PR builds on that work rather than replacing it. (f29dbd3ebd1c)
  • Feature history: Recent relevant history includes the latest split-check commit plus merged smoke taxonomy/profile and QA suite speed work by RomneyDa. (0455bddb70a0)

Likely related people:

  • RomneyDa: Authored the latest split-check commit and related merged QA smoke profile, speed, fail-on-QA, and taxonomy cleanup work. (role: recent QA/CI area contributor; confidence: high; commits: 0455bddb70a0, f29dbd3ebd1c, 9666db607eb5; files: .github/workflows/ci.yml, docs/ci.md, test/scripts/ci-workflow-guards.test.ts)
  • vincentkoc: Current-main shallow blame and recent history point to CI topology and guard-test work on the same workflow surface. (role: recent CI workflow contributor; confidence: medium; commits: 20293036cab5; files: .github/workflows/ci.yml, test/scripts/ci-workflow-guards.test.ts)
  • Solvely-Colin: Authored the original PR and merged prerequisite QA Lab evidence infrastructure, so they have useful context beyond only opening this branch. (role: adjacent QA Lab evidence contributor; confidence: medium; commits: 591313e80a5e, c677424edba0, d5a27b0b9684; files: extensions/qa-lab/src, scripts/qa, qa/scenarios)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. labels Jun 17, 2026
@clawsweeper clawsweeper Bot mentioned this pull request Jun 18, 2026
Merged
@Solvely-Colin
Solvely-Colin marked this pull request as ready for review June 18, 2026 01:10
Copilot AI review requested due to automatic review settings June 18, 2026 01:10
@Solvely-Colin
Solvely-Colin marked this pull request as draft June 18, 2026 01:12
Copilot AI reviewed Jun 18, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a manually-dispatched GitHub Actions workflow for running QA Lab smoke evidence on Ubuntu runners, and introduces a QA Lab Evidence Archive surface (server endpoints + UI) for browsing qa-evidence.json bundles and declared artifacts.

Changes:

  • Add dispatch-only QA Lab Smoke Profile workflow to run smoke-ci (or a specific scenario_id) and upload .artifacts/qa-e2e/.
  • Add QA Lab Evidence Archive API routes (/api/evidence, /api/evidence/artifact) plus an evidence gallery model for resolving/previewing declared artifacts.
  • Add QA Lab web UI tab + styling + tests for Evidence Archive rendering and UX Matrix mini-grid linking.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
.github/workflows/qa-lab-smoke-profile.yml New workflow_dispatch action to run QA Lab smoke profile/suite and upload evidence artifacts.
scripts/check-no-raw-channel-fetch.mjs Updates allowlisted raw fetch callsite line numbers for QA Lab web app.
extensions/qa-lab/src/lab-server.ts Adds evidence gallery API endpoints and artifact streaming with headers.
extensions/qa-lab/src/lab-server.test.ts Adds server test coverage for evidence artifact HEAD/GET behavior and auth errors.
extensions/qa-lab/src/evidence-gallery.ts New evidence gallery model + artifact resolution/containment logic + previews + UX Matrix context parsing.
extensions/qa-lab/src/evidence-gallery.test.ts New tests for evidence gallery model building, containment, symlink defense, and producer context detection.
extensions/qa-lab/web/src/app.ts Adds Evidence Archive state, URL parsing, loading behavior, and UI event handlers.
extensions/qa-lab/web/src/ui-render.ts Adds Evidence Archive tab rendering, filtering, matrix mini-grid, and “Open evidence” affordance.
extensions/qa-lab/web/src/ui-render.test.ts New render tests for evidence status tone mapping and UX Matrix cell linking behavior.
extensions/qa-lab/web/src/styles.css Adds Evidence Archive layout/styles and evidence-focused shell mode.

Comment thread extensions/qa-lab/src/evidence-gallery.ts Outdated
Comment thread extensions/qa-lab/web/src/app.ts
Comment thread .github/workflows/qa-lab-smoke-profile.yml Outdated
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Jun 18, 2026
View reviewed changes

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d02aab0854

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/qa-lab/src/evidence-gallery.ts Outdated
@Solvely-Colin
Solvely-Colin force-pushed the codex/qa-lab-smoke-profile-action branch from d02aab0 to 5ad7ca5 Compare June 18, 2026 01:40
@Solvely-Colin Solvely-Colin changed the title ci: add QA Lab smoke profile dispatch ci: draft QA Lab smoke profile dispatch wiring Jun 18, 2026
@Solvely-Colin
Solvely-Colin force-pushed the codex/qa-lab-smoke-profile-action branch from 5ad7ca5 to 7510ca3 Compare June 18, 2026 01:42
@Solvely-Colin

Solvely-Colin commented Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Updated this draft PR to do the cleanup ClawSweeper requested where possible:

  • Rebased directly on current origin/main so ci: run QA smoke profile in CI #94291 is workflow-only now; it no longer carries the qa-lab: add evidence artifact gallery #94283 Evidence Gallery runtime/UI stack.
  • Updated the title/body/risk section to make the scope explicit: draft QA Lab smoke-profile workflow wiring, not a runtime/UI change.
  • Re-ran local proof on head 7510ca31bc:
    • git diff --check origin/main...HEAD passed
    • PATH="/Users/solvely/.local/bin:/opt/homebrew/bin:$PATH" node scripts/check-workflows.mjs passed
    • go run github.com/rhysd/actionlint/cmd/[email protected] .github/workflows/qa-lab-smoke-profile.yml passed
  • Re-attempted the requested real GitHub workflow_dispatch; GitHub still returns the expected default-branch limitation because the new workflow file is not on default yet:
    • HTTP 404: workflow qa-lab-smoke-profile.yml not found on the default branch

This keeps the PR draft and documents the remaining maintainer decision: keep this as a dedicated workflow with first dispatch proof after/default-branch, or fold the command shape into an existing smoke-ci workflow so runner proof can happen pre-merge.

@clawsweeper

clawsweeper Bot commented Jun 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot mentioned this pull request Jun 18, 2026
Merged
@Solvely-Colin
Solvely-Colin force-pushed the codex/qa-lab-smoke-profile-action branch 2 times, most recently from a668f3b to 21bebd4 Compare June 18, 2026 14:29
@openclaw-barnacle openclaw-barnacle Bot added the scripts Repository scripts label Jun 24, 2026
@RomneyDa
RomneyDa marked this pull request as ready for review June 24, 2026 06:39
@openclaw-barnacle openclaw-barnacle Bot removed the scripts Repository scripts label Jun 24, 2026
@RomneyDa RomneyDa changed the title ci: shard QA smoke profile in CI ci: run QA smoke profile in existing CI shard Jun 24, 2026
@RomneyDa RomneyDa mentioned this pull request Jun 24, 2026
Merged
Solvely-Colin and others added 23 commits June 24, 2026 09:31
@Solvely-Colin @RomneyDa
ci: add qa lab smoke profile dispatch
fae1cc5
@Solvely-Colin @RomneyDa
ci: prove qa lab smoke profile on prs
e48dd0c
@Solvely-Colin @RomneyDa
ci: preserve manual qa lab profile dispatch
8ae7ed8
@RomneyDa
ci: run qa lab smoke profile on pull requests
d4af562
@RomneyDa
ci: keep QA smoke mock lane bounded
ae431ac
@RomneyDa
ci: run QA smoke PR proof through crabline
761dd1e
@RomneyDa
ci: keep mock QA timeouts on caller fallbacks
14588dc
@RomneyDa
ci: prebuild QA smoke runtime
73cf6c2
@RomneyDa
ci: delegate smoke QA evidence workflow
7dc167a
@RomneyDa
ci: trust release branch smoke evidence refs
dd68883
@RomneyDa
ci: trim smoke evidence workflow comments
58ed776
@RomneyDa
ci: align smoke evidence wrapper with QA profile contract
8f3ce53
@RomneyDa
ci: keep smoke profile evidence mock-only
63c4c54
@RomneyDa
ci: make smoke profile evidence manual
4518ea0
@RomneyDa
ci: shard qa smoke profile in ci
14618ac
@RomneyDa
ci: drop qa-channel-only smoke shard
bc968f2
@RomneyDa
ci: derive qa smoke shards from taxonomy
8b56dc4
@RomneyDa
ci: keep qa smoke planner legacy-safe
b59e3f7
@RomneyDa
ci: enforce qa smoke shard failures
86cf14d
@RomneyDa
ci: run qa smoke in existing fast shard
5ca0c17
@RomneyDa
ci: opt qa smoke into crabline concurrency
8efddf9
@RomneyDa
test(ci): align qa smoke guard with taxonomy cleanup
99c73b4
@RomneyDa
ci: split qa smoke into dedicated check
0455bdd
@github-actions github-actions Bot mentioned this pull request Jun 24, 2026
Open
@clawsweeper clawsweeper Bot mentioned this pull request Jun 29, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

Assignees

No one assigned

Labels

docs Improvements or additions to documentation maintainer Maintainer-authored PR merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. P2 Normal backlog priority with limited blast radius. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: S status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Solvely-Colin @RomneyDa