qa-lab: support script-backed evidence scenarios#94276
Merged
RomneyDa merged 3 commits intoJun 17, 2026
Merged
Conversation
openclaw-barnacle
Bot
added
triage: needs-real-behavior-proof
Solvely-Colin
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new QA Lab native scenario execution kind (script) so QA scenarios can run producer scripts (via Node + tsx) and import their qa-evidence.json output into suite evidence/coverage reporting.
Changes:
- Extend scenario catalog + runner types to support
scenario.execution.kind: scriptwith optionalargstoken expansion (${outputDir},${scenarioId}). - Import and normalize producer-written
qa-evidence.jsonfor script scenarios, merging it into the suite evidence output (and preserving fallback runner evidence on script failure). - Update coverage reporting and tests to recognize
scriptscenarios as native evidence sources.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| qa/scenarios/index.yaml | Docs/comments updated to include script as an execution kind. |
| extensions/qa-lab/src/test-file-scenario-runner.ts | Adds script runner support, token expansion, producer evidence import/normalization, and evidence merge logic. |
| extensions/qa-lab/src/test-file-scenario-runner.test.ts | Adds coverage for running script scenarios and importing producer evidence (including failure cases). |
| extensions/qa-lab/src/suite-launch.runtime.test.ts | Updates mocked scenario execution kind union to include script. |
| extensions/qa-lab/src/scorecard-taxonomy.ts | Extends native evidence kind union to include script. |
| extensions/qa-lab/src/scenario-catalog.ts | Adds script to execution schema with optional args. |
| extensions/qa-lab/src/evidence-summary.ts | Adds buildScriptEvidenceSummary helper. |
| extensions/qa-lab/src/coverage-report.ts | Includes script in scenario match grouping order. |
| extensions/qa-lab/src/coverage-report.test.ts | Adds coverage test asserting script evidence fulfills coverage IDs. |
…mplify artifact path resolution
crh-code
pushed a commit
to crh-code/openclaw
that referenced
this pull request
Jun 18, 2026
* qa: add script scenario execution kind * fix(qa-lab): carry suite profile into script producer evidence and simplify artifact path resolution * fix(qa-lab): keep out-of-repo producer artifacts absolute to avoid ../ traversal refs --------- Co-authored-by: Dallin Romney <[email protected]>
github-actions Bot
pushed a commit
to Desicool/openclaw
that referenced
this pull request
Jun 18, 2026
* qa: add script scenario execution kind * fix(qa-lab): carry suite profile into script producer evidence and simplify artifact path resolution * fix(qa-lab): keep out-of-repo producer artifacts absolute to avoid ../ traversal refs --------- Co-authored-by: Dallin Romney <[email protected]>
This was referenced Jun 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
QA Lab already knows how to run flow, Vitest, and Playwright-backed scenarios. The UX Matrix split needs one more reusable layer first: a native
scriptexecution kind for scenario producers that create their ownqa-evidence.jsonartifacts.This PR is intentionally only that infrastructure layer.
What changed
scenario.execution.kind: scriptto the QA scenario catalog schema${outputDir}and${scenarioId}argument token expansionqa-evidence.jsonartifacts into the suite evidence outputSplit context
This is PR 1 split out from the closed combined UX Matrix spike PR #94273.
Follow-up slices should stay separate:
This PR does not add the UX Matrix scenario, evidence gallery UI changes, dashboard script, or GitHub Action.
Real behavior proof
9df6cecb3595fb2f0cee897e241ddc0e1878f4ab.pnpm openclaw qa coverage --helpandpnpm openclaw qa coverage --match playwright.Supplemental validation
node scripts/run-vitest.mjs extensions/qa-lab/src/test-file-scenario-runner.test.ts extensions/qa-lab/src/coverage-report.test.ts extensions/qa-lab/src/scenario-catalog.test.ts extensions/qa-lab/src/suite-launch.runtime.test.tspnpm qa:lab:buildnode scripts/run-tsgo.mjs -p tsconfig.extensions.json --incremental falsepnpm lint --threads=8pnpm run lint:extensions:bundledgit diff --check --cached