Summary

What problem does this PR solve?

• Adds a QA Lab-backed UX Matrix evidence producer for key OpenClaw journey stages across Web UI, CLI, TUI, Android, iOS/iPad simulator, macOS app, watchOS, and live gateway lanes.
• Adds a QA Lab Evidence view for the generated qa-evidence.json and suite artifact directories so reviewers can inspect entries, coverage IDs, screenshots, GIF/video, logs, machine-validation files, scorecards, release ledgers, and runner plans.
• Adds a dedicated UX Matrix QA workflow that runs the QA Lab scenario on existing OpenClaw runner infrastructure and uploads the evidence bundle.

Why does this matter now?

• Maintainer feedback suggested folding the script into existing QA Lab + evidence as a scenario or test and using the existing evidence artifact source/path structure for screenshots and related proof.
• This turns the local UX Matrix work into a repeatable QA Lab scenario that can run locally and in GitHub Actions instead of staying as a standalone local artifact.

What is the intended outcome?

• pnpm openclaw qa suite --scenario ux-matrix-evidence-dashboard produces a QA evidence summary plus UX Matrix artifacts.
• QA Lab can load that evidence bundle and show the matrix, proof entries, screenshots/GIF/video links, logs, machine-validation files, and runner plans.
• The matrix remains honest: only executed cells produce QA evidence entries; unexecuted cells remain proof gaps with actionable runner/coverage metadata.

What is intentionally out of scope?

• This PR does not fully execute Android emulator, iOS/iPad simulator, macOS app, watchOS, TUI PTY, or live gateway proof cells inside the UX Matrix producer yet.
• This PR does not claim proof for cells that did not emit artifacts in the run.
• This PR does not change CHANGELOG.md.

What does success look like?

• The QA Lab scenario runs and imports the producer's qa-evidence.json.
• The Evidence tab loads the suite artifact directory or qa-evidence.json and stays responsive while showing the matrix and proof drilldowns.
• The generated matrix has 135 cells with 4 currently executed proof-backed cells and actionable runner plans for the remaining proof gaps.

What should reviewers focus on?

• Whether the QA Lab scenario integration is the right home for this producer.
• Whether the qa-evidence.json import/merge behavior is conservative and does not overclaim coverage.
• Whether the new UX Matrix QA workflow fits existing OpenClaw CI patterns.
• Whether the UI evidence drilldown is useful without making large media previews lock the page.

AI-assisted: yes, implemented with Codex. I understand the code path and kept the evidence/coverage distinction explicit.

Linked context

Which issue does this close?

Closes #

Which issues, PRs, or discussions are related?

Related #

Was this requested by a maintainer or owner?

• Yes, maintainer discussion asked how this script could fold into QA Lab + evidence as a new scenario or vitest/playwright/script style test, whether it needs new coverage IDs or can cover existing taxonomy.yaml IDs, and whether the UI generation could work with QA evidence-formatted output.

Real behavior proof (required for external PRs)

• Behavior or issue addressed:

  • Adds repeatable UX Matrix QA evidence generation and a QA Lab evidence dashboard for reviewing matrix proof artifacts.

• Real environment tested:

  • Local Mac checkout, branch codex/ux-matrix-local-dashboard.
  • Clean proof run at SHA 8ac4664d162be1117487a2077c8e7a5220338888.
  • QA Lab evidence model loaded the generated evidence locally after the rebase.

• Exact steps or command run after this patch:

  • pnpm openclaw qa suite --provider-mode mock-openai --scenario ux-matrix-evidence-dashboard --output-dir .artifacts/qa-e2e/ux-matrix-rebased-proof --concurrency 1 --fast
  • pnpm ux:matrix -- --once --recording-ms 2000
  • Evidence model load check for /Users/solvely/Documents/OpenClaw-baselines/worktrees/openclaw-ux-matrix-tool/.artifacts/ux-matrix/20260617T201115Z-8ac4664d16/qa-evidence.json

• Evidence after fix (screenshot, recording, terminal capture, console output, redacted runtime log, linked artifact, or copied live output):

  • QA Lab scenario suite report:
    • .artifacts/qa-e2e/ux-matrix-rebased-proof/qa-suite-report.md
    • Result: Passed: 1, Failed: 0
  • QA Lab suite evidence:
    • .artifacts/qa-e2e/ux-matrix-rebased-proof/qa-evidence.json
    • Result: 4 entries, all pass
  • Clean UX Matrix artifact:
    • .artifacts/ux-matrix/20260617T201115Z-8ac4664d16/qa-evidence.json
    • .artifacts/ux-matrix/20260617T201115Z-8ac4664d16/matrix.json
    • .artifacts/ux-matrix/20260617T201115Z-8ac4664d16/scorecard.md
  • Proof artifacts emitted:
    • .artifacts/ux-matrix/20260617T201115Z-8ac4664d16/surfaces/web-ui/stages/first-run/screenshot.png
    • .artifacts/ux-matrix/20260617T201115Z-8ac4664d16/surfaces/web-ui/stages/first-run/recording.gif
    • .artifacts/ux-matrix/20260617T201115Z-8ac4664d16/surfaces/web-ui/stages/first-run/recording.webm
    • .artifacts/ux-matrix/20260617T201115Z-8ac4664d16/surfaces/web-ui/stages/gateway-connected-healthy/screenshot.png
    • .artifacts/ux-matrix/20260617T201115Z-8ac4664d16/surfaces/web-ui/stages/gateway-connected-healthy/recording.gif
    • .artifacts/ux-matrix/20260617T201115Z-8ac4664d16/surfaces/web-ui/stages/gateway-connected-healthy/recording.webm
    • CLI logs and machine-validation files under .artifacts/ux-matrix/20260617T201115Z-8ac4664d16/surfaces/cli/stages/
  • Copied live output from the clean scorecard:
    • Run: 20260617T201115Z-8ac4664d16
    • Status: pass
    • Dirty checkout: false
    • pass: 4
    • proof-gap: 131
    • CI-schedulable gaps: 90
    • Manual/live gaps: 15
  • Evidence model load check result:
    • loaded: true
    • entries: 4
    • cells: 135
    • gaps: 131

• Observed result after fix:

  • QA Lab scenario runner imports producer QA evidence artifacts.
  • Evidence tab loads the generated evidence and maps the 135-cell matrix to the 4 proof-backed evidence entries.
  • Web UI cells include screenshot, GIF, WebM, log, and machine-validation artifacts.
  • CLI cells include log and machine-validation artifacts.
  • Proof gaps remain visible as gaps with coverage IDs and runner plans instead of being reported as failures.

• What was not tested:

  • GitHub Actions execution of .github/workflows/ux-matrix-qa.yml has not run yet because the branch has not been pushed/opened as a PR.
  • Android emulator, Apple simulator, macOS app UI, watchOS, TUI PTY, and live gateway cells were not executed as UX Matrix proof cells in this local run.
  • Live provider/gateway credentials were not used.

• Proof limitations or environment constraints:

  • The default local run intentionally avoids launching Android emulator, Apple simulators, Docker, or real gateway/live credentials to avoid local OOM pressure and private-state risk.
  • Running the QA suite locally printed existing local config/plugin-version warnings from shared machine state, but the scenario completed successfully and produced suite evidence.

• Before evidence (optional but encouraged):

  • Before this PR, QA Lab did not have the UX Matrix producer/evidence dashboard path; the UX Matrix was local-script/output oriented and the evidence UI could not load or drill into this matrix proof bundle.

Tests and validation

Which commands did you run?

• pnpm openclaw qa suite --provider-mode mock-openai --scenario ux-matrix-evidence-dashboard --output-dir .artifacts/qa-e2e/ux-matrix-rebased-proof --concurrency 1 --fast
• pnpm ux:matrix -- --once --recording-ms 2000
• node scripts/run-vitest.mjs extensions/qa-lab/src/evidence-gallery.test.ts extensions/qa-lab/web/src/ui-render.test.ts extensions/qa-lab/src/scenario-catalog.test.ts extensions/qa-lab/src/coverage-report.test.ts extensions/qa-lab/src/test-file-scenario-runner.test.ts
• pnpm qa:lab:build
• node scripts/run-tsgo.mjs -p tsconfig.extensions.json --incremental false
• git diff --check

What regression coverage was added or updated?

• Added/updated QA Lab tests for:
  • script scenario execution and producer QA evidence import
  • UX Matrix producer context loading
  • Evidence gallery matrix/media/log drilldowns
  • UI rendering of matrix cells, proof gaps, and deferred heavy media previews
  • scenario catalog and coverage report handling for the UX Matrix scenario
  • script producer failure propagation so failed/blocked imported producer evidence cannot leave the QA suite green
  • evidence artifact download bounds so the gallery can serve declared artifacts and UX Matrix producer context files without serving arbitrary repo files

What failed before this fix, if known?

• The local prototype did not exist as a QA Lab scenario and was not runnable through QA Lab/GitHub runner paths.
• Earlier local UI iterations could lock up on evidence media previews and had Evidence page scroll issues; this branch includes fixes that defer heavy previews and restore page scrolling.

If no test was added, why not?

• Tests were added and updated.

Risk checklist

Did user-visible behavior change? (Yes/No)

Yes. QA Lab gains an Evidence tab workflow for UX Matrix artifacts and a new UX Matrix scenario.

Did config, environment, or migration behavior change? (Yes/No)

No config or migration behavior change. A new GitHub Actions workflow was added.

Did security, auth, secrets, network, or tool execution behavior change? (Yes/No)

Yes, tool execution behavior changes in a bounded QA context: QA Lab can run a declared execution.kind: script scenario and import its QA evidence artifacts. The UX Matrix producer redacts sensitive token-like values from logs and the workflow uses mock provider mode.

What is the highest-risk area?

• Importing producer-generated evidence into suite-level QA evidence without overclaiming coverage or linking proof gaps as if they were executed tests.
• Rendering large media artifacts in the Evidence UI without hurting responsiveness.

How is that risk mitigated?

• Only executed UX Matrix cells are emitted as QA evidence entries.
• Proof gaps remain matrix cells with status proof-gap, coverage IDs, and runner plans, but no evidence entry link.
• Heavy GIF/WebM previews are deferred to artifact links instead of embedded inline by default.
• Local proof covered the QA Lab scenario path, direct producer path, evidence model loading, focused tests, build, typecheck, and diff check.

Current review state

What is the next action?

• Push branch and open PR.
• Let .github/workflows/ux-matrix-qa.yml run on GitHub and attach/upload its generated artifact.

What is still waiting on author, maintainer, CI, or external proof?

• Waiting on GitHub Actions proof for the new workflow.
• Waiting on future slices to execute/import Android, Apple simulator, macOS, watchOS, TUI, and live gateway proof cells as actual UX Matrix entries.

Which bot or reviewer comments were addressed?

• Local Codex review finding addressed: restricted /api/evidence/artifact to declared evidence artifacts plus narrow UX Matrix producer context files.
• Local Codex re-review findings addressed: producer evidence status propagation, ux:matrix --once failure exit code, missing Chromium-not-runnable logs.txt, and background dashboard run unhandled rejection behavior.
• Local Codex re-review finding addressed: script scenarios now preserve producer evidence when a script exits nonzero after writing qa-evidence.json.
• Local Codex re-review finding addressed: evidence artifact resolution now prefers artifacts beside the loaded evidence summary before repo-root fallback paths, with regression coverage for path collisions.
• Local Codex re-review finding addressed: blocked and skipped evidence entries now map to styled badge/dot tones in the Evidence UI.
• Local Codex re-review finding addressed: producer context previews now reject symlinked files whose real path leaves the allowed evidence/repo roots.