test(qa): link local e2e tests into QA coverage#97150
Conversation
|
Codex review: needs maintainer review before merge. Reviewed June 26, 2026, 10:19 PM ET / 02:19 UTC. Summary PR surface: Other +160. Total +160 across 7 files. Reproducibility: not applicable. This PR is QA coverage bookkeeping rather than a product bug. The native scenario runner and all referenced test files are present on current main. Review metrics: 1 noteworthy metric.
Stored data model Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Security Review detailsBest possible solution: Land the YAML wrappers after exact-head CI/QA checks finish, or revise any wrapper whose local test is not accepted as primary maturity evidence. Do we have a high-confidence way to reproduce the issue? Not applicable; this PR is QA coverage bookkeeping rather than a product bug. The native scenario runner and all referenced test files are present on current main. Is this the best way to solve the issue? Yes; the existing QA contract supports small YAML wrappers with execution.kind: vitest and execution.path, so linking existing tests is narrower than duplicating QA flow logic. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 552ec2b49d4a. Label changesLabel justifications:
Evidence reviewedPR surface: Other +160. Total +160 across 7 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
What Problem This Solves
Existing e2e tests already prove several maturity coverage IDs, but QA scenario inventory did not link them as native QA evidence. That left the coverage score lower than the available proof.
Why This Change Was Made
This adds small native QA scenario YAML wrappers for deterministic local Vitest e2e tests instead of duplicating test logic. Each wrapper uses an existing taxonomy coverage ID as primary evidence and points to the existing proof file through
scenario.execution.path.This intentionally skips candidates where the existing proof was too indirect for primary coverage, including the Anthropic auth-profile row, watchOS generic exec approval, voice-call generic RPC/tool coverage, release-candidate checklist parsing, and live/provider media rows.
User Impact
QA maturity inventory can now account for 8 additional existing primary coverage IDs when these native scenarios pass:
automation.active-hoursautomation.heartbeat-schedulingui.assistant-media-ticketsui.browser-talk-start-stopchannels.native-command-session-targetclawhub.skill-installscli.channel-pickerdocker.composeEvidence
pnpm test extensions/qa-lab/src/scenario-catalog.test.ts extensions/qa-lab/src/coverage-report.test.ts extensions/qa-lab/src/scorecard-evidence.test.tspnpm openclaw qa suite --provider-mode mock-openai --scenario heartbeat-active-hours --scenario control-ui-assistant-media-tickets --scenario browser-talk-start-stop --scenario native-command-session-target --scenario clawhub-skill-installs --scenario cli-channel-picker --scenario docker-compose-setup --output-dir .artifacts/qa-e2e/local-vitest-wrapper-pr1-rerunpnpm openclaw qa coverage --json > .artifacts/qa-e2e/local-vitest-wrapper-pr1-rerun/qa-coverage.json.agents/skills/autoreview/scripts/autoreview --mode localgit diff --check -- qa/scenariosThe final QA evidence run had 7 passing entries covering the 8 primary IDs above.
qa coverage --jsonreportedscenarioCount: 136andprimaryCoverageIdCount: 113after this change.