test: add QA Lab UX Matrix evidence scenario#94306
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a Vitest-backed QA Lab scenario that produces UX Matrix evidence bundles and extends QA Lab’s server + web UI to load, browse, and stream declared evidence artifacts (including merged/suite evidence and blocked/skipped status handling).
Changes:
- Add
ux-matrix-evidence-dashboardQA scenario and a Vitest producer test that writes UX Matrix artifacts +qa-evidence.json. - Introduce Evidence Gallery model/types and QA Lab server endpoints to load evidence and serve declared artifacts.
- Extend QA Lab web UI with an “Evidence Archive” tab, filters, and producer-context rendering.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/check-no-raw-channel-fetch.mjs | Updates allowlisted callsite line numbers for qa-lab web app. |
| qa/scenarios/ui/ux-matrix-evidence-dashboard.yaml | Adds new QA scenario definition targeting a Vitest producer with sidecar evidence summary. |
| extensions/qa-lab/shared/evidence-gallery-types.ts | Defines shared view-model types for the Evidence Archive UI. |
| extensions/qa-lab/src/evidence-gallery.ts | Implements evidence bundle loading, producer-context discovery, and artifact path containment/authorization. |
| extensions/qa-lab/src/evidence-gallery.test.ts | Adds tests for gallery model building, UX Matrix producer detection (full/slim), and artifact containment. |
| extensions/qa-lab/src/lab-server.ts | Adds /api/evidence and /api/evidence/artifact endpoints and maps gallery errors to HTTP responses. |
| extensions/qa-lab/src/lab-server.test.ts | Tests artifact HEAD/GET streaming, headers, and declared/outside artifact rejections. |
| extensions/qa-lab/web/src/ui-render.ts | Adds Evidence Archive rendering (filters, entry list, artifact cards, producer mini-grid). |
| extensions/qa-lab/web/src/styles.css | Adds Evidence Archive layout + styling and evidence-focused shell layout. |
| extensions/qa-lab/web/src/app.ts | Adds Evidence Archive state, URL param handling, and client-side evidence loading/actions. |
| extensions/qa-lab/web/src/ui-render.test.ts | Tests evidence status tone mapping and UX Matrix cell linking/deferred previews. |
| extensions/qa-lab/src/test-file-scenario-runner.ts | Supports scenario-produced evidence sidecars and merges them into suite evidence output. |
| extensions/qa-lab/src/test-file-scenario-runner.test.ts | Tests sidecar evidence usage, stale sidecar cleanup, and failure fallback behavior. |
| extensions/qa-lab/src/suite.ts | Extends suite scenario status to include skip. |
| extensions/qa-lab/src/suite-launch.runtime.ts | Maps blocked/skipped native evidence to suite skip while preserving evidence.status detail. |
| extensions/qa-lab/src/suite-launch.runtime.test.ts | Adds coverage for blocked native evidence mapping to skipped suite status. |
| extensions/qa-lab/src/scenario-catalog.test.ts | Updates catalog expectations and asserts YAML config for the new scenario. |
| extensions/qa-lab/src/ux-matrix-evidence.test.ts | Adds the Vitest UX Matrix evidence producer that writes artifacts and validates gallery loadability. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e92898cedc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
@clawsweeper re-review |
|
Codex review: needs maintainer review before merge. Reviewed June 18, 2026, 4:38 PM ET / 20:38 UTC. Summary PR surface: Tests +71, Other +671. Total +742 across 4 files. Reproducibility: not applicable. This PR adds a QA Lab evidence scenario rather than fixing a current-main bug. The changed behavior can be exercised with Review metrics: 1 noteworthy metric.
Stored data model Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Next step before merge
Security Review detailsBest possible solution: Land this narrow scenario-producer slice if maintainers want the UX Matrix scenario, while keeping Evidence Archive UI and workflow dispatch wiring in their separate related PRs. Do we have a high-confidence way to reproduce the issue? Not applicable; this PR adds a QA Lab evidence scenario rather than fixing a current-main bug. The changed behavior can be exercised with Is this the best way to solve the issue? Yes; using the merged AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 61b116d5972c. Label changesLabel justifications:
Evidence reviewedPR surface: Tests +71, Other +671. Total +742 across 4 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
6ff3c68 to
5bd80b6
Compare
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
710885e to
30b8ae2
Compare
* test: add qa lab ux matrix script scenario * fix(qa-lab): annotate UX Matrix producer catch callback as unknown for oxlint --------- Co-authored-by: Dallin Romney <[email protected]>
* test: add qa lab ux matrix script scenario * fix(qa-lab): annotate UX Matrix producer catch callback as unknown for oxlint --------- Co-authored-by: Dallin Romney <[email protected]>
Summary
ux-matrix-evidence-dashboardas a QA Lab script-backed scenario using the existing qa-lab: support script-backed evidence scenarios #94276 producer contract:execution.kind: scriptwrites/imports producer-ownedqa-evidence.json.scripts/qa/ux-matrix-evidence-producer.ts, which writes a UX Matrix evidence bundle with coverage IDs, logs, screenshot/video artifacts where local Playwright can produce them, and blocked evidence when local prerequisites are unavailable.Scope Notes
execution.config.evidenceSummaryPathor Vitest sidecar evidence API is added.producer-artifact-fixtureHTML/PNG/WebM is producer/artifact proof only; it is explicitly not the QA Lab Evidence Archive UI.911a2aafba4d0142ca9ce1a28aaaedefae54da36while loading this PR's generated UX Matrixqa-evidence.json.Verification
git diff --checkpnpm exec oxfmt --check --threads=1 scripts/qa/ux-matrix-evidence-producer.ts qa/scenarios/ui/ux-matrix-evidence-dashboard.yaml extensions/qa-lab/src/scenario-catalog.test.ts extensions/qa-lab/src/test-file-scenario-runner.test.tsnode scripts/run-vitest.mjs extensions/qa-lab/src/scenario-catalog.test.ts extensions/qa-lab/src/test-file-scenario-runner.test.ts --reporter=verboseOPENCLAW_BUILD_PRIVATE_QA=1 pnpm openclaw qa suite --scenario ux-matrix-evidence-dashboard --output-dir .artifacts/qa-e2e/ux-matrix-script-proof --allow-failures.agents/skills/autoreview/scripts/autoreviewVisual Proof
These embedded visuals are GitHub-hosted proof from the actual #94283 QA Lab Evidence Archive UI loading this PR's UX Matrix
qa-evidence.json. They are not the producer artifact fixture.Fallback raw files:
Evidence
Generated UX Matrix evidence from this branch:
.artifacts/qa-e2e/ux-matrix-script-proof/qa-evidence.json.artifacts/qa-e2e/ux-matrix-script-proof/qa-suite-summary.json.artifacts/qa-e2e/ux-matrix-script-proof/qa-suite-report.md.artifacts/qa-e2e/ux-matrix-script-proof/script/ux-matrix-evidence-dashboard/qa-evidence.jsonReal QA Lab Evidence Archive UI proof, captured from the actual #94283 UI stack loading the UX Matrix evidence bundle:
OPENCLAW_BUILD_PRIVATE_QA=1 pnpm openclaw qa ui --port 58083 --embedded-gateway disabledQA_LAB_UI_URL=http://127.0.0.1:58083 node .artifacts/qa-e2e/ux-matrix-script-proof/real-evidence-archive-ui/capture-real-evidence-archive-ui.mjs(Playwright); it opens/evidence?path=.artifacts%2Fqa-e2e%2Fux-matrix-script-proof%2Fqa-evidence.json, selectsux-matrix.control-ui.screenshot-artifact, then selectsux-matrix.cli.entrypoint-helpand expandsPreview log.Solvely-Colin/pr-proof-assets@e60c91bc39061561be9e245eea56b216630a2a8f/openclaw-pr-94306/Producer/artifact fixture proof only, not UI proof:
.artifacts/qa-e2e/ux-matrix-script-proof/script/ux-matrix-evidence-dashboard/surfaces/qa-lab/stages/producer-artifact-fixture/producer-artifact-fixture.html.artifacts/qa-e2e/ux-matrix-script-proof/script/ux-matrix-evidence-dashboard/surfaces/qa-lab/stages/producer-artifact-fixture/producer-artifact-fixture.png.artifacts/qa-e2e/ux-matrix-script-proof/script/ux-matrix-evidence-dashboard/surfaces/qa-lab/stages/producer-artifact-fixture/producer-artifact-fixture.webmObserved local evidence result: scenario status
pass, 3 evidence entries, 7 artifacts, 0 missing artifacts when loaded through the #94283 Evidence Archive API/UI.Real behavior proof
Behavior addressed: QA Lab now has a repeatable UX Matrix scenario that uses the existing script producer contract and emits consumable QA evidence entries/artifacts.
Real environment tested: Local macOS source checkout, branch
codex/qa-lab-ux-matrix-scenario, commit5bd80b6620088953145686ff579c9336330e312b; real UI proof captured from #94283 stack911a2aafba4d0142ca9ce1a28aaaedefae54da36.Exact steps or command run after this patch: See Verification, Visual Proof, and Evidence above.
Evidence after fix: The scenario writes suite evidence, producer-owned
qa-evidence.json, logs, screenshots, WebM producer fixture proof, and GitHub-hosted embedded real #94283 Evidence Archive UI screenshot/GIF proof.Observed result after fix: Focused QA Lab tests passed;
qa suite --scenario ux-matrix-evidence-dashboardproduced 3 passing UX Matrix evidence entries: producer artifact fixture, control UI screenshot artifact, and CLI entrypoint help.What was not tested: GitHub Actions/profile wiring was intentionally not added or tested. The real Evidence Archive UI proof depends on #94283's gallery UI stack being available; this PR does not include that UI implementation.