Skip to content

test: fold lifecycle and package proof into QA Lab#93114

Merged
RomneyDa merged 10 commits into
mainfrom
script-to-test-conversion
Jun 17, 2026
Merged

test: fold lifecycle and package proof into QA Lab#93114
RomneyDa merged 10 commits into
mainfrom
script-to-test-conversion

Conversation

@RomneyDa

@RomneyDa RomneyDa commented Jun 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

  • Base RFC Fix path traversal vulnerability in media server #10 script-elimination slice for plugin lifecycle and package-artifact proof.
  • Delete the plugin lifecycle matrix probe/sweep scripts and the old test/scripts probe test.
  • Fold lifecycle assertions into QA Lab-owned e2e coverage under test/e2e/qa-lab/plugins, with a Docker-safe runtime helper for the full lifecycle container lane.
  • Move package artifact Docker proof into test/e2e/qa-lab/runtime/package-openclaw-for-docker.e2e.test.ts and add matching QA scenario YAML.
  • Update QA scenario catalog expectations, changed-test routing, and Docker harness mounts so the migrated QA/e2e files are selected and runnable.

Stack Notes

Coverage Notes

  • Adds QA scenario evidence for existing taxonomy IDs plugins.lifecycle and docker-e2e-package-artifact-generation.
  • Keeps lifecycle evidence executable through both QA Lab scenario execution and the full Docker lifecycle lane.
  • Does not claim the OTEL, gateway, or channel coverage IDs; those are covered by the stacked follow-ups.

Verification

  • node scripts/run-vitest.mjs test/e2e/qa-lab/plugins/plugin-lifecycle-probe.e2e.test.ts test/scripts/docker-build-helper.test.ts test/scripts/test-projects.test.ts extensions/qa-lab/src/scenario-catalog.test.ts
  • git diff --check
  • bash -n scripts/e2e/plugin-lifecycle-matrix-docker.sh
  • OPENCLAW_CURRENT_PACKAGE_TGZ=/tmp/does-not-exist.tgz npx --yes tsx test/e2e/qa-lab/plugins/plugin-lifecycle-probe-runtime.ts --lifecycle-matrix confirmed the lifecycle runtime entrypoint transforms and starts; it then fails only because the test tarball path is intentionally missing.
  • Full Docker lifecycle lane on Testbox-through-Crabbox:
    • provider/id: blacksmith-testbox / tbx_01kvbnanbwqsmvvjwwn3td3snf
    • Actions run: 27718658493
    • command: corepack pnpm test:docker:plugin-lifecycle-matrix
    • result: passed; lifecycle phases covered install, inspect, disable, enable, upgrade, downgrade, and missing-code uninstall.
  • .agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main
    • result: clean, no accepted/actionable findings.

Known local warning during older QA commands: existing state migration warning for .openclaw/plugins/installs.json.migrated already being present.

@clawsweeper

clawsweeper Bot commented Jun 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 17, 2026, 4:43 PM ET / 20:43 UTC.

Summary
The PR moves package Docker and plugin lifecycle script-test coverage into QA Lab scenario-backed e2e tests, adds matching QA scenario YAML, updates changed-test routing, and routes the plugin lifecycle Docker lane through the QA e2e file.

PR surface: Tests +509, Other -196. Total +313 across 13 files.

Reproducibility: not applicable. as a user bug report; this is a PR review of a QA/script refactor. The relevant check is source-review plus current-head QA and Docker automation proof.

Review metrics: 2 noteworthy metrics.

  • QA scenario scope: 2 in current diff, 5 still described in body. The final branch split three migrations into stacked PRs, so the public verification text should match the actual merge surface.
  • Bare Docker harness mounts: 2 added, 0 removed. The shared harness now exposes QA Lab tests and shared test helpers to every bare Docker harness run, which is relevant automation surface before merge.

Stored data model
Persistent data-model change detected: migration/backfill/repair: qa/scenarios/plugins/plugin-lifecycle-probe.yaml, serialized state: scripts/e2e/lib/plugin-lifecycle-matrix/probe.mjs, serialized state: test/e2e/qa-lab/plugins/plugin-lifecycle-probe.e2e.test.ts, serialized state: test/scripts/plugin-lifecycle-probe.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Refresh the PR body so the summary and verification match the current two-scenario diff.
  • Record current-head proof for pnpm openclaw qa suite --scenario package-openclaw-for-docker --scenario plugin-lifecycle-probe and pnpm test:docker:plugin-lifecycle-matrix.

Risk before merge

  • [P1] The shared bare Docker harness now mounts QA/test helper trees and executes the plugin lifecycle lane through a TS e2e file, so the current-head plugin lifecycle Docker lane should be treated as required pre-merge proof.
  • [P1] The PR body still describes five migrated scenarios and lists commands for split-out scenarios and an old --probe path, so reviewers need refreshed verification text before relying on the proof.

Maintainer options:

  1. Refresh current-head automation proof (recommended)
    Update the PR body to the final two-scenario scope and record current-head QA suite plus plugin lifecycle Docker E2E proof before merge.
  2. Accept proof drift temporarily
    Maintainers could merge with the stale body, but they would be knowingly relying on CI and local review instead of the listed verification commands.

Next step before merge

  • [P2] No narrow ClawSweeper code repair is indicated; the remaining work is maintainer/author proof refresh for an automation-sensitive, maintainer-labeled PR.

Security
Cleared: No concrete security or supply-chain issue was found; the diff changes tests, QA metadata, and read-only Docker harness mounts without adding dependencies, secrets, workflow permissions, or publishing hooks.

Review details

Best possible solution:

Keep the split QA scenario migration, then land only after exact current-head QA and Docker lifecycle verification replaces the stale five-scenario proof text.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a user bug report; this is a PR review of a QA/script refactor. The relevant check is source-review plus current-head QA and Docker automation proof.

Is this the best way to solve the issue?

Yes, moving script-owned evidence into QA Lab is the right owner boundary for this cleanup. The mergeable form should keep the current split scope and refresh proof for the exact head before landing.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 20534c57b7e8.

Label changes

Label changes:

  • add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The external contributor proof gate does not apply because this is a member/maintainer-labeled PR, though its current-head verification text still needs normal maintainer refresh before merge.
  • remove rating: 🧂 unranked krab: Current PR rating is rating: 🦪 silver shellfish, so this older rating label is no longer current.
  • remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

  • P3: This is an internal QA/script coverage cleanup with limited user-facing blast radius.
  • merge-risk: 🚨 automation: The diff changes Docker E2E harness mounts and the plugin lifecycle matrix execution path, which can break validation automation even when ordinary tests are green.
  • rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The external contributor proof gate does not apply because this is a member/maintainer-labeled PR, though its current-head verification text still needs normal maintainer refresh before merge.
Evidence reviewed

PR surface:

Tests +509, Other -196. Total +313 across 13 files.

View PR surface stats
Area Files Added Removed Net
Source 0 0 0 0
Tests 6 623 114 +509
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 7 61 257 -196
Total 13 684 371 +313

What I checked:

Likely related people:

  • RomneyDa: Recent merged history shows this handle authored the YAML scenario catalog conversion and QA suite execution work that this PR extends. (role: recent QA scenario owner; confidence: high; commits: fef839407995, a3e9dfee0e39, 561b293c7a96; files: extensions/qa-lab/src/scenario-catalog.test.ts, qa/scenarios/index.yaml, extensions/qa-lab/src/test-file-scenario-runner.ts)
  • vincentkoc: Recent commit history for the plugin lifecycle matrix and Docker harness points to this handle as the most active adjacent owner for the automation path this PR changes. (role: recent Docker E2E adjacent owner; confidence: high; commits: cfaae7761dc0, 71cb60706b82, a3d5e5bc7247; files: scripts/e2e/plugin-lifecycle-matrix-docker.sh, scripts/e2e/lib/plugin-lifecycle-matrix/sweep.sh, scripts/lib/docker-e2e-package.sh)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. labels Jun 15, 2026
@openclaw-barnacle openclaw-barnacle Bot added scripts Repository scripts size: M and removed size: S labels Jun 15, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 15, 2026
@openclaw-barnacle openclaw-barnacle Bot added docker Docker and sandbox tooling size: L and removed size: M labels Jun 15, 2026
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. labels Jun 15, 2026
This was referenced Jun 15, 2026
Merged
Merged
Merged
Closed
@RomneyDa
RomneyDa force-pushed the script-to-test-conversion branch 2 times, most recently from 4fa026e to afa188f Compare June 17, 2026 18:47
@clawsweeper clawsweeper Bot removed the rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. label Jun 17, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. and removed status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 17, 2026
@RomneyDa
RomneyDa force-pushed the script-to-test-conversion branch from e0571de to 6965711 Compare June 17, 2026 19:56
@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 17, 2026
@RomneyDa RomneyDa changed the title test: fold script coverage into QA scenarios test: fold lifecycle and package proof into QA Lab Jun 17, 2026
@blacksmith-sh

This comment has been minimized.

Sign in to view
@RomneyDa
RomneyDa merged commit 0a6736a into main Jun 17, 2026
170 checks passed
@RomneyDa
RomneyDa deleted the script-to-test-conversion branch June 17, 2026 21:22
@github-actions github-actions Bot mentioned this pull request Jun 17, 2026
Open
crh-code pushed a commit to crh-code/openclaw that referenced this pull request Jun 18, 2026
@RomneyDa @crh-code
test: fold lifecycle and package proof into QA Lab (openclaw#93114)
d754862 
* test: fold script coverage into qa scenarios

* test: migrate script checks into qa e2e

* test: point qa code refs at migrated e2e

* test: fold plugin lifecycle probe into qa e2e

* test: use shared temp dirs in plugin lifecycle probe

* test: fold plugin lifecycle sweep into qa lab

* test: trim lifecycle docker text assertions

* test: keep followup script conversions split

* test: make lifecycle docker runner script-safe

* test: update changed helper routing expectation
This was referenced Jun 18, 2026
Closed
Merged
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 18, 2026
@RomneyDa
test: fold lifecycle and package proof into QA Lab (openclaw#93114)
16b62f7 
* test: fold script coverage into qa scenarios

* test: migrate script checks into qa e2e

* test: point qa code refs at migrated e2e

* test: fold plugin lifecycle probe into qa e2e

* test: use shared temp dirs in plugin lifecycle probe

* test: fold plugin lifecycle sweep into qa lab

* test: trim lifecycle docker text assertions

* test: keep followup script conversions split

* test: make lifecycle docker runner script-safe

* test: update changed helper routing expectation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

docker Docker and sandbox tooling extensions: qa-lab maintainer Maintainer-authored PR merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. scripts Repository scripts size: XL status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@RomneyDa