test(qa): add qa run --qa-profile and unified output summary/evidence by RomneyDa · Pull Request #91587 · openclaw/openclaw

RomneyDa · 2026-06-09T03:38:55Z

Summary

Adds pnpm openclaw qa run --qa-profile smoke-ci|release as the taxonomy-backed QA profile entry point.
Resolves profile, surface, and category selectors from taxonomy.yaml, filters mapped scenarios against the selected provider lane, and dispatches the resulting scenario set through qa suite.
Makes qa suite execute mixed scenario sets directly: flow scenarios, execution.kind: vitest, and execution.kind: playwright can run in one selected suite.
Writes one top-level qa-evidence.json, qa-suite-summary.json, and qa-suite-report.md for the selected suite/profile, while keeping native Vitest/Playwright per-scenario logs under partition directories.
Keeps the older qa run self-check behavior when no --qa-profile is provided, and preserves root-profile parsing for all qa run --profile <name> values.
Updates QA docs to describe the unified suite artifacts and profile command flow.

Notes

qa-evidence.json is the canonical evidence artifact for all scenario execution kinds.
qa-suite-summary.json remains for existing parity/confidence/reporting consumers, but it now includes the unified selected scenario set rather than being flow-only.
Profile dispatch does not force provider-incompatible scenarios. For example, smoke-ci defaults to mock-openai and filters out scenarios requiring live-frontier.
Native qa-vitest-report.md / qa-playwright-report.md subreports were removed intentionally. They were added recently, were not adopted by release scripts or other consumers, and their useful data is already covered by the top-level suite report, canonical evidence, and per-scenario logs.
Native log references in suite artifacts are repo-relative, not absolute local checkout paths.

Verification

git diff --check
node scripts/run-vitest.mjs extensions/qa-lab/src/test-file-scenario-runner.test.ts extensions/qa-lab/src/suite-launch.runtime.test.ts extensions/qa-lab/src/cli.runtime.test.ts extensions/qa-lab/src/cli.test.ts src/cli/profile.test.ts
pnpm check:test-types
pnpm check:docs
.agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main found one profile/provider filtering issue; fixed in b3e846a5c0.
.agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main found one absolute native log path issue; fixed in e7cc83821f.
ClawSweeper flagged the qa run --profile root-profile collision; fixed by moving the QA selector to --qa-profile in 356e863190.

clawsweeper · 2026-06-09T03:45:55Z

Codex review: needs changes before merge. Reviewed June 14, 2026, 9:05 PM ET / 01:05 UTC.

Summary
The PR adds taxonomy-backed qa run --qa-profile dispatch, mixed flow/Vitest/Playwright qa suite execution, unified top-level QA artifacts, and matching QA docs/tests while preserving root --profile parsing.

PR surface: Source +432, Tests +402, Docs +27. Total +861 across 13 files.

Reproducibility: yes. Source inspection shows a mixed run copies child flow evidence entries into the top-level evidence file unchanged while the flow suite emits output-local basename artifact paths.

Review metrics: 2 noteworthy metrics.

qa run CLI mode surface: 10 profile-only flags added, 1 self-check-only flag guarded. The PR expands an existing command with mode-specific flags, so explicit validation is important for operator scripts.
QA report artifact names: 2 native subreports removed, 3 top-level suite artifacts retained. The PR changes the current-main evidence/report artifact contract that proof tooling and local scripts may consume.

Stored data model
Persistent data-model change detected: database schema: extensions/qa-lab/src/suite-launch.runtime.test.ts, serialized state: extensions/qa-lab/src/suite-launch.runtime.test.ts, serialized state: extensions/qa-lab/src/test-file-scenario-runner.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🌊 off-meta tidepool
Patch quality: 🦐 gold shrimp
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P2] Rebase merged child evidence artifact paths and add a regression with non-empty flow evidence entries.
Keep the artifact-consolidation decision visible for maintainer acceptance before landing.

Risk before merge

[P1] The PR changes an existing QA CLI command and consolidates current-main QA artifact filenames, so maintainers should consciously accept the compatibility shape for scripts and evidence consumers before merge.
[P1] The unified top-level qa-evidence.json can currently mispoint flow-suite artifact references in mixed runs, which can break proof capture or downstream evidence readers even when the suite itself passes.

Maintainer options:

Rebase Child Evidence Paths (recommended)
Rewrite merged child evidence entries so flow, Vitest, and Playwright partition artifacts point at the correct repo-relative paths in the top-level qa-evidence.json, and add a non-empty evidence regression test.
Accept Artifact Surface Consolidation
Maintainers may accept dropping the main-only native subreport filenames once the canonical top-level evidence/report/summary paths are internally consistent and documented.

Copy recommended automerge instruction

@clawsweeper automerge

Special instructions:
Rebase merged child-suite evidence artifact paths in unified `qa suite` output so entries copied from flow/Vitest/Playwright partition evidence point at the correct repo-relative partition artifacts, and add a focused `suite-launch.runtime.test.ts` regression with non-empty child evidence entries.

Next step before merge

[P2] A narrow automated repair can rebase merged child evidence artifact paths and add a focused regression test without deciding the broader artifact-consolidation policy.

Security
Cleared: The diff is limited to QA CLI/runtime/docs/tests and does not change workflows, dependency sources, lockfiles, package scripts, secrets handling, or downloaded code execution paths.

Review findings

[P2] Rebase merged flow evidence artifact paths — extensions/qa-lab/src/suite-launch.runtime.ts:308

Review details

Best possible solution:

Land the QA profile runner after rebasing child partition evidence artifact paths into the unified evidence file and after maintainers accept the current-main artifact consolidation.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection shows a mixed run copies child flow evidence entries into the top-level evidence file unchanged while the flow suite emits output-local basename artifact paths.

Is this the best way to solve the issue?

No, not yet. The --qa-profile split and unified artifact direction are maintainable, but the evidence merge needs path rebasing before the top-level artifact can be the canonical proof surface.

Full review comments:

[P2] Rebase merged flow evidence artifact paths — extensions/qa-lab/src/suite-launch.runtime.ts:308
Mixed suites now copy each child partition's qa-evidence.json entries into the top-level evidence file unchanged. Flow-suite evidence records its report and summary as output-local basenames, so after this merge those entries point at the top-level unified qa-suite-report.md/qa-suite-summary.json instead of the flow/ partition artifacts. Rebase child evidence artifact paths before merging and add a non-empty flow evidence test so proof consumers open the right artifacts.
Confidence: 0.88

Overall correctness: patch is incorrect
Overall confidence: 0.86

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 870ec6dee291.

Label changes

Label changes:

add merge-risk: 🚨 automation: The PR changes QA evidence artifact generation, and the current merged evidence path issue can mislead proof capture or automation that opens artifact references.

Label justifications:

P2: This is bounded QA CLI/evidence infrastructure work with a concrete pre-merge correctness issue but no user-runtime outage.
merge-risk: 🚨 compatibility: The PR changes an existing CLI command's accepted flags and consolidates QA artifact filenames that current-main scripts or evidence consumers may reference.
merge-risk: 🚨 automation: The PR changes QA evidence artifact generation, and the current merged evidence path issue can mislead proof capture or automation that opens artifact references.
rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Not applicable: This is a MEMBER-authored maintainer PR, so the external-contributor real behavior proof gate does not apply; the body lists focused validation commands.

Evidence reviewed

PR surface:

Source +432, Tests +402, Docs +27. Total +861 across 13 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	6	652	220	+432
Tests	5	449	47	+402
Docs	2	29	2	+27
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	13	1130	269	+861

Acceptance criteria:

[P1] node scripts/run-vitest.mjs extensions/qa-lab/src/suite-launch.runtime.test.ts extensions/qa-lab/src/cli.test.ts extensions/qa-lab/src/cli.runtime.test.ts src/cli/profile.test.ts.
[P1] pnpm check:test-types.
[P1] git diff --check.

What I checked:

Repository policy read: Read the full root AGENTS.md plus scoped docs and extensions guides; CLI compatibility, docs accuracy, plugin-boundary, protected-label, and PR review-depth guidance apply to this PR. (AGENTS.md:1, 870ec6dee291)
Gitcrawl unavailable, live GitHub used: The required cached gitcrawl path was attempted first, but the binary is unavailable in this checkout; live gh and local source/history were used instead.
Protected maintainer PR: Live PR metadata and prompt context show this PR is open, MEMBER-authored, and labeled maintainer, so conservative cleanup must keep it open rather than close it. (aea44c92f73a)
Previous finding fixed at latest head: The latest head adds validateQaRunMode, rejects profile-only flags without --qa-profile, rejects self-check --output in profile mode, and has focused CLI tests for those cases. (extensions/qa-lab/src/cli.ts:113, aea44c92f73a)
Root profile parsing preserved: At PR head, --profile is still stripped as a root profile for qa run, while --qa-profile is preserved for the QA command parser. (src/cli/profile.ts:37, aea44c92f73a)
Profile dispatch uses taxonomy and provider filtering: runQaProfileCommand resolves profiles/categories from the taxonomy report, filters selected scenarios with scenarioMatchesQaProviderLane, and dispatches the scenario IDs through runQaSuiteCommand. (extensions/qa-lab/src/cli.runtime.ts:622, aea44c92f73a)

Likely related people:

RomneyDa: Authored the recent merged QA scorecard taxonomy/profile work and this PR builds directly on that taxonomy and evidence-mapping surface. (role: QA scorecard/profile implementation owner; confidence: high; commits: d8b3e523ff7b, a3e9dfee0e39; files: taxonomy.yaml, extensions/qa-lab/src/scorecard-taxonomy.ts, extensions/qa-lab/src/coverage-report.ts)
steipete: Commit history shows prior merged work on root profile parsing and the original interactive QA Lab suite runner, both adjacent to this PR's CLI/profile compatibility boundary. (role: root CLI and QA suite history contributor; confidence: medium; commits: 751c7f32a55e, 350238d40226; files: src/cli/profile.ts, src/cli/profile.test.ts, extensions/qa-lab/src/suite-launch.runtime.ts)
Ayaan Zaidi: Local blame on the sampled current-main QA launcher, QA run registration, and root profile parser lines points to the checkout's grafted base commit, but the commit topic is broad so this is a weak routing signal. (role: recent current-line author; confidence: low; commits: 65e6d9c98c; files: extensions/qa-lab/src/suite-launch.runtime.ts, extensions/qa-lab/src/cli.ts, src/cli/profile.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

RomneyDa · 2026-06-10T08:12:08Z

@clawsweeper re-review

clawsweeper · 2026-06-10T08:12:11Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/27262700915
Updated: 2026-06-10T08:26:38.290Z

RomneyDa · 2026-06-14T08:33:50Z

@clawsweeper re-review

clawsweeper · 2026-06-14T08:33:53Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Superseded
Detail: A newer re-review for this item started before this run finished, so GitHub cancelled this older run. Check the latest ClawSweeper run for the current result.
Run: https://github.com/openclaw/clawsweeper/actions/runs/27493669690
Updated: 2026-06-14T08:50:44.133Z

RomneyDa · 2026-06-14T08:46:10Z

@clawsweeper re-review

RomneyDa · 2026-06-15T00:34:47Z

@clawsweeper re-review

clawsweeper · 2026-06-15T00:34:50Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/27517237243
Updated: 2026-06-15T00:43:34.078Z

…penclaw#91587) * test(qa): add mapped qa run profiles * test(qa): document mapped profile runner * test(qa): validate run profiles from mapping * test(qa): preserve root profile parsing * test(qa): simplify taxonomy profile dispatch * test(qa): align tool coverage CLI expectation * test(qa): fix profile dispatch fixture type * test(qa): share profile runner option types * test(qa): split shared cli runner options * test(qa): unify profile suite artifacts * fix(qa): filter profile scenarios by provider lane * test(qa): drop native scenario subreports * fix(qa): keep native log refs repo-relative * fix(cli): preserve qa run root profile parsing * fix(qa): avoid qa profile flag collision * fix(qa): reject profile flags without qa profile

openclaw-barnacle Bot added cli CLI command changes extensions: qa-lab size: M maintainer Maintainer-authored PR labels Jun 9, 2026

clawsweeper Bot added the rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. label Jun 9, 2026

RomneyDa force-pushed the dallin/coverage-taxonomy-validation branch from 0e1f24b to 9bbbdb5 Compare June 9, 2026 04:05

RomneyDa force-pushed the dallin/qa-run-profile-dispatcher branch from 5582d66 to c6dbb66 Compare June 9, 2026 04:06

RomneyDa force-pushed the dallin/coverage-taxonomy-validation branch from 5d12f72 to 4b769f4 Compare June 9, 2026 18:45

RomneyDa force-pushed the dallin/qa-run-profile-dispatcher branch from c6dbb66 to 20cf355 Compare June 9, 2026 18:47

clawsweeper Bot mentioned this pull request Jun 9, 2026

Add QA scorecard taxonomy validation #91500

Merged

clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Jun 9, 2026

RomneyDa force-pushed the dallin/coverage-taxonomy-validation branch from 4b769f4 to 7b521df Compare June 9, 2026 23:24

RomneyDa force-pushed the dallin/qa-run-profile-dispatcher branch from 20cf355 to fd7d762 Compare June 9, 2026 23:25

RomneyDa force-pushed the dallin/coverage-taxonomy-validation branch 2 times, most recently from 76133af to ce7f048 Compare June 10, 2026 08:00

RomneyDa force-pushed the dallin/qa-run-profile-dispatcher branch from fd7d762 to dcb0e9a Compare June 10, 2026 08:02

openclaw-barnacle Bot added docs Improvements or additions to documentation size: L and removed size: M labels Jun 10, 2026

clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jun 10, 2026

openclaw-barnacle Bot added size: XL and removed size: L labels Jun 14, 2026

fix(qa): filter profile scenarios by provider lane

b3e846a

RomneyDa changed the title ~~test(qa): add mapped qa run profiles~~ test(qa): add qa run --profile and unified output summary/evidence Jun 14, 2026

RomneyDa added 3 commits June 14, 2026 01:50

test(qa): drop native scenario subreports

7a734d4

fix(qa): keep native log refs repo-relative

e7cc838

fix(cli): preserve qa run root profile parsing

32f8179

clawsweeper Bot mentioned this pull request Jun 14, 2026

docs: link maturity scorecard docs #91483

Merged

fix(qa): avoid qa profile flag collision

356e863

clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jun 15, 2026

fix(qa): reject profile flags without qa profile

aea44c9

RomneyDa mentioned this pull request Jun 15, 2026

test(reply): seed channel fixtures for dedupe tests #93104

Merged

clawsweeper Bot added the merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. label Jun 15, 2026

RomneyDa merged commit e8db9c3 into main Jun 15, 2026
171 checks passed

RomneyDa deleted the dallin/qa-run-profile-dispatcher branch June 15, 2026 01:08

RomneyDa mentioned this pull request Jun 15, 2026

test(reply): preserve telegram dedupe fallback #93107

Merged

github-actions Bot mentioned this pull request Jun 15, 2026

📡 Upstream Digest — 2026-06-15 02:54 UTC curtismercier/openclaw-mods#1081

Open

clawsweeper Bot mentioned this pull request Jun 15, 2026

fix(acp): preserve timeout progress summaries #91479

Open

RomneyDa mentioned this pull request Jun 15, 2026

Implement RFC 0007: E2E and QA Lab scorecard consolidation #91883

Closed

RomneyDa changed the title ~~test(qa): add qa run --profile and unified output summary/evidence~~ test(qa): add qa run --qa-profile and unified output summary/evidence Jun 16, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

test(qa): add qa run --qa-profile and unified output summary/evidence#91587

test(qa): add qa run --qa-profile and unified output summary/evidence#91587
RomneyDa merged 16 commits into
mainfrom
dallin/qa-run-profile-dispatcher

RomneyDa commented Jun 9, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 9, 2026 •

edited

Loading

Uh oh!

RomneyDa commented Jun 10, 2026

Uh oh!

clawsweeper Bot commented Jun 10, 2026 •

edited

Loading

Uh oh!

RomneyDa commented Jun 14, 2026

Uh oh!

clawsweeper Bot commented Jun 14, 2026 •

edited

Loading

Uh oh!

RomneyDa commented Jun 14, 2026

Uh oh!

RomneyDa commented Jun 15, 2026

Uh oh!

clawsweeper Bot commented Jun 15, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

RomneyDa commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Notes

Verification

Uh oh!

clawsweeper Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RomneyDa commented Jun 10, 2026

Uh oh!

clawsweeper Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RomneyDa commented Jun 14, 2026

Uh oh!

clawsweeper Bot commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RomneyDa commented Jun 14, 2026

Uh oh!

RomneyDa commented Jun 15, 2026

Uh oh!

clawsweeper Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

RomneyDa commented Jun 9, 2026 •

edited

Loading

clawsweeper Bot commented Jun 9, 2026 •

edited

Loading

clawsweeper Bot commented Jun 10, 2026 •

edited

Loading

clawsweeper Bot commented Jun 14, 2026 •

edited

Loading

clawsweeper Bot commented Jun 15, 2026 •

edited

Loading