Skip to content

test(qa): link local e2e tests into QA coverage#97150

Merged
RomneyDa merged 1 commit into
mainfrom
dallin/qa-link-local-vitest-e2e
Jun 28, 2026
Merged

test(qa): link local e2e tests into QA coverage#97150
RomneyDa merged 1 commit into
mainfrom
dallin/qa-link-local-vitest-e2e

Conversation

@RomneyDa

Copy link
Copy Markdown
Member

What Problem This Solves

Existing e2e tests already prove several maturity coverage IDs, but QA scenario inventory did not link them as native QA evidence. That left the coverage score lower than the available proof.

Why This Change Was Made

This adds small native QA scenario YAML wrappers for deterministic local Vitest e2e tests instead of duplicating test logic. Each wrapper uses an existing taxonomy coverage ID as primary evidence and points to the existing proof file through scenario.execution.path.

This intentionally skips candidates where the existing proof was too indirect for primary coverage, including the Anthropic auth-profile row, watchOS generic exec approval, voice-call generic RPC/tool coverage, release-candidate checklist parsing, and live/provider media rows.

User Impact

QA maturity inventory can now account for 8 additional existing primary coverage IDs when these native scenarios pass:

  • automation.active-hours
  • automation.heartbeat-scheduling
  • ui.assistant-media-tickets
  • ui.browser-talk-start-stop
  • channels.native-command-session-target
  • clawhub.skill-installs
  • cli.channel-picker
  • docker.compose

Evidence

  • pnpm test extensions/qa-lab/src/scenario-catalog.test.ts extensions/qa-lab/src/coverage-report.test.ts extensions/qa-lab/src/scorecard-evidence.test.ts
  • pnpm openclaw qa suite --provider-mode mock-openai --scenario heartbeat-active-hours --scenario control-ui-assistant-media-tickets --scenario browser-talk-start-stop --scenario native-command-session-target --scenario clawhub-skill-installs --scenario cli-channel-picker --scenario docker-compose-setup --output-dir .artifacts/qa-e2e/local-vitest-wrapper-pr1-rerun
  • pnpm openclaw qa coverage --json > .artifacts/qa-e2e/local-vitest-wrapper-pr1-rerun/qa-coverage.json
  • .agents/skills/autoreview/scripts/autoreview --mode local
  • git diff --check -- qa/scenarios

The final QA evidence run had 7 passing entries covering the 8 primary IDs above. qa coverage --json reported scenarioCount: 136 and primaryCoverageIdCount: 113 after this change.

@RomneyDa RomneyDa changed the title Link local e2e tests into QA coverage test(qa): link local e2e tests into QA coverage Jun 27, 2026
@clawsweeper

clawsweeper Bot commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 26, 2026, 10:19 PM ET / 02:19 UTC.

Summary
Adds seven QA scenario YAML wrappers that link existing local Vitest e2e tests to eight primary QA maturity coverage IDs.

PR surface: Other +160. Total +160 across 7 files.

Reproducibility: not applicable. This PR is QA coverage bookkeeping rather than a product bug. The native scenario runner and all referenced test files are present on current main.

Review metrics: 1 noteworthy metric.

  • Primary coverage links: 8 primary coverage IDs linked by 7 wrappers. This is the maturity-score surface changed by the PR, separate from ordinary changed-file counts.

Stored data model
Persistent data-model change detected: migration/backfill/repair: qa/scenarios/config/cli-channel-picker.yaml, vector/embedding metadata: qa/scenarios/ui/control-ui-assistant-media-tickets.yaml. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Wait for the exact-head CI and QA smoke jobs to complete before merge.

Risk before merge

  • [P1] Exact-head CI and QA smoke jobs were still in progress during live inspection, and this read-only review did not rerun the PR commands locally.

Maintainer options:

  1. Decide the mitigation before merge
    Land the YAML wrappers after exact-head CI/QA checks finish, or revise any wrapper whose local test is not accepted as primary maturity evidence.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • Maintainer-labeled MEMBER PR with no narrow ClawSweeper repair identified; it needs normal maintainer review after exact-head checks finish.

Security
Cleared: The diff adds QA scenario YAML metadata only and does not change workflows, dependencies, lockfiles, secrets handling, install scripts, or runtime code.

Review details

Best possible solution:

Land the YAML wrappers after exact-head CI/QA checks finish, or revise any wrapper whose local test is not accepted as primary maturity evidence.

Do we have a high-confidence way to reproduce the issue?

Not applicable; this PR is QA coverage bookkeeping rather than a product bug. The native scenario runner and all referenced test files are present on current main.

Is this the best way to solve the issue?

Yes; the existing QA contract supports small YAML wrappers with execution.kind: vitest and execution.path, so linking existing tests is narrower than duplicating QA flow logic.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 552ec2b49d4a.

Label changes

Label justifications:

  • P3: This is low-risk QA coverage bookkeeping that links existing deterministic tests into maturity inventory without changing runtime behavior.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The external-contributor proof gate does not apply to this maintainer-labeled MEMBER PR; the PR body still includes terminal command evidence for the QA suite and coverage run.
Evidence reviewed

PR surface:

Other +160. Total +160 across 7 files.

View PR surface stats
Area Files Added Removed Net
Source 0 0 0 0
Tests 0 0 0 0
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 7 160 0 +160
Total 7 160 0 +160

What I checked:

  • Repository policy applied: Root AGENTS.md was read fully; it routes PR review through OpenClaw maintainer workflow and says QA scenario sources under qa/scenarios must be YAML. (AGENTS.md:149, 552ec2b49d4a)
  • Live PR state: Live PR metadata shows an open, mergeable PR at head ff62890 with the protected maintainer label and no review comments. (ff6289049084)
  • Current main does not already contain the wrappers: Searching current main for the new scenario IDs produced no matches under qa/scenarios or QA Lab source, so the PR is not redundant on main. (552ec2b49d4a)
  • Scenario YAML contract: The QA scenario pack explicitly documents native test scenarios with scenario.execution.kind: vitest/playwright/script plus scenario.execution.path, which matches the PR shape. (qa/scenarios/index.yaml:29, 552ec2b49d4a)
  • Catalog schema and discovery: Current main validates vitest/playwright/script execution kinds and recursively discovers nested qa/scenarios YAML files, so the added files are in the consumed catalog path. (extensions/qa-lab/src/scenario-catalog.ts:85, 552ec2b49d4a)
  • Vitest scenario runner: The native runner treats vitest scenarios as test-file scenarios, invokes scripts/run-vitest.mjs on scenario.execution.path, and carries scenario coverage/docs/code refs into evidence targets. (extensions/qa-lab/src/test-file-scenario-runner.ts:101, 552ec2b49d4a)

Likely related people:

  • RomneyDa: Authored the merged native Vitest/Playwright QA scenario runner work in Run Vitest and Playwright scenarios from qa suite #92606 and recent QA evidence/coverage commits touching the same runner and catalog surfaces. (role: feature owner and recent area contributor; confidence: high; commits: 61a528690de6, f568a4abd3b6, 67b0433cf84c; files: extensions/qa-lab/src/test-file-scenario-runner.ts, extensions/qa-lab/src/suite-launch.runtime.ts, extensions/qa-lab/src/scenario-catalog.ts)
  • Solvely-Colin: Authored the merged script-backed QA evidence scenario PR that modified the same catalog, runner, evidence, and scorecard paths. (role: adjacent contributor; confidence: medium; commits: 9df6cecb3595; files: extensions/qa-lab/src/test-file-scenario-runner.ts, extensions/qa-lab/src/evidence-summary.ts, extensions/qa-lab/src/scenario-catalog.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. labels Jun 27, 2026
@RomneyDa
RomneyDa merged commit 78f7de0 into main Jun 28, 2026
119 of 124 checks passed
@RomneyDa
RomneyDa deleted the dallin/qa-link-local-vitest-e2e branch June 28, 2026 01:34
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 28, 2026
QiuYuang pushed a commit to QiuYuang/openclaw that referenced this pull request Jul 1, 2026
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 1, 2026
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extensions: qa-lab maintainer Maintainer-authored PR P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant