Skip to content

improve(qa): execute runtime scenarios through Docker#99705

Merged
RomneyDa merged 2 commits into
mainfrom
dallin/wave1-docker-runtime
Jul 4, 2026
Merged

improve(qa): execute runtime scenarios through Docker#99705
RomneyDa merged 2 commits into
mainfrom
dallin/wave1-docker-runtime

Conversation

@RomneyDa

@RomneyDa RomneyDa commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What Problem This Solves

Resolves a QA coverage gap where six runtime and plugin scenarios reported helper-level Vitest proof instead of exercising their existing Docker runtime topology and product-facing APIs.

Why This Change Was Made

Routes the six scenario definitions through the shared Docker E2E lane adapter, preserving the existing lane implementations and focused helper tests. The native web-search assertion scenario intentionally shares the same runtime lane as the minimal web-search scenario because that lane emits and validates both the successful native Responses request and the minimal-reasoning rejection evidence.

The plugin lifecycle runtime probe also keeps its temporary-directory cleanup local so the package-installed bare Docker image does not need the Vitest-only test helper dependency.

User Impact

No product behavior changes. Maintainers now receive real container/runtime evidence for OpenAI-compatible chat tools, OpenAI native web search, OpenWebUI compatibility, plugin lifecycle operations, and packaged bundled-plugin install/uninstall coverage when these QA scenarios run.

Evidence

  • Exact-head targeted Docker run 28688381529 passed at 37783c3502e427b47a24370af24fb4c50ef0c625:
    • openai-chat-tools
    • openai-web-search-minimal (covers both minimal and native-request assertion scenarios)
    • openwebui
    • plugin-lifecycle-matrix
    • bundled-plugin-install-uninstall (all scheduler-expanded shards)
  • The targeted workflow reused docker-e2e-package from run 28568249540 because current main and the initial exact-head run both fail package preparation at the same pre-existing write-plugin-sdk-entry-dts build step. The source run's package preparation job succeeded; this PR changes no package contents or Docker lane implementations.
  • node scripts/run-vitest.mjs test/e2e/qa-lab/plugins/plugin-lifecycle-probe.e2e.test.ts extensions/qa-lab/src/scenario-catalog.test.ts test/e2e/qa-lab/runtime/docker-e2e-lane.fixture.test.ts --reporter=verbose — 47 tests passed after the final rebase.
  • Targeted oxfmt --check and git diff --check passed.
  • Blacksmith Testbox changed-file gate passed on tbx_01kwn62778hx14fh400ahcjq78 (Actions run 28688249628).
  • Structured Codex autoreview reported no accepted/actionable findings with 0.98 correctness confidence.

Known CI gap: exact-head CI run 28688380283 failed QA Smoke CI because Docker script scenarios and QA flow partitions concurrently mutate/read the shared dist/ build. The single failed-job rerun reproduced the race (dist/index.js missing, then a generated chunk missing). Current main QA Smoke passes without these newly script-backed scenarios. This shared QA-runner orchestration fix is intentionally not mixed into this PR and has been requested as a separate micro-fix.

@clawsweeper

clawsweeper Bot commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 3, 2026, 9:07 PM ET / 01:07 UTC.

Summary
The PR remaps six QA scenario YAML entries from Vitest helper tests to the existing Docker lane script adapter and localizes plugin lifecycle probe temp-dir cleanup for runtime execution.

PR surface: Tests +37, Other +33. Total +70 across 8 files.

Reproducibility: not applicable. as a user-facing bug reproduction; this is a QA automation PR. Source inspection confirms current main uses Vitest execution for representative scenarios, the PR-head maps them to Docker script lanes, and exact-head CI reproduces the separate QA Smoke orchestration failure tracked in #99734.

Review metrics: 1 noteworthy metric.

  • QA scenario execution remap: 6 scenarios changed from Vitest helper execution to Docker script lane execution. This is the maintainer-visible automation behavior change that needs targeted Docker and QA Smoke validation before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

Risk before merge

Maintainer options:

  1. Fix or green the QA Smoke orchestration path (recommended)
    Resolve [Bug]: QA Smoke scripts can invalidate dist used by flow Gateways #99734 or rerun QA Smoke successfully with these script-backed scenarios before merge.
  2. Accept targeted Docker proof deliberately
    Maintainers can choose to merge on the passing targeted Docker lane run while tracking the QA Smoke race as a separate follow-up.
  3. Pause until the smoke race lands
    Keep this PR open if the QA Smoke failure should remain a hard gate for changing scenario execution in the smoke profile.

Next step before merge

  • [P2] The protected maintainer label and current QA Smoke automation failure make this a maintainer merge decision rather than an automated repair candidate.

Security
Cleared: The diff changes QA scenario metadata, a QA test, and local test-runtime cleanup only; it adds no dependencies, workflows, permissions, secrets handling, or third-party code source.

Review details

Best possible solution:

Land this PR after the QA Smoke orchestration race is fixed or consciously accepted, keeping the scenario mappings on the existing script runner and Docker lane registry rather than adding a parallel QA path.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a user-facing bug reproduction; this is a QA automation PR. Source inspection confirms current main uses Vitest execution for representative scenarios, the PR-head maps them to Docker script lanes, and exact-head CI reproduces the separate QA Smoke orchestration failure tracked in #99734.

Is this the best way to solve the issue?

Yes, the PR uses the existing script scenario runner and shared Docker lane registry, which is the narrow maintainable shape for this coverage gap. The safer merge path is to fix or explicitly accept the separate QA Smoke scheduler race rather than changing these lane mappings back to helper tests.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 1f499b01c425.

Label changes

Label changes:

  • add merge-risk: 🚨 automation: The diff changes QA scenario execution and exact-head QA Smoke currently fails from a shared-dist orchestration race exposed by script-backed Docker scenarios.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The external-contributor proof gate does not apply to this member-authored maintainer-labeled PR; nevertheless, the PR body and Actions run include exact-head targeted Docker lane proof.

Label justifications:

  • P3: This is a focused maintainer QA coverage improvement with no product runtime behavior change.
  • merge-risk: 🚨 automation: The diff changes QA scenario execution and exact-head QA Smoke currently fails from a shared-dist orchestration race exposed by script-backed Docker scenarios.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The external-contributor proof gate does not apply to this member-authored maintainer-labeled PR; nevertheless, the PR body and Actions run include exact-head targeted Docker lane proof.
Evidence reviewed

PR surface:

Tests +37, Other +33. Total +70 across 8 files.

View PR surface stats
Area Files Added Removed Net
Source 0 0 0 0
Tests 2 43 6 +37
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 6 55 22 +33
Total 8 98 28 +70

What I checked:

Likely related people:

  • RomneyDa: GitHub path history shows recent merged work expanding the Docker lane adapter, enforcing QA scenario ownership, and maintaining the same scenario-catalog and QA suite surfaces touched here. (role: recent QA Lab and Docker adapter contributor; confidence: high; commits: fd3f354f4626, 677c8ff8466d, 561b293c7a96; files: test/e2e/qa-lab/runtime/docker-e2e-lane.fixture.ts, extensions/qa-lab/src/scenario-catalog.test.ts, extensions/qa-lab/src/suite-launch.runtime.ts)
  • vincentkoc: GitHub path history shows repeated lifecycle matrix, Docker E2E, timeout, and resource-limit work on the plugin lifecycle runtime and script surfaces reused by this PR. (role: plugin lifecycle Docker lane contributor; confidence: high; commits: ea45950a9d8e, b306745f64fe, 604aa301899f; files: scripts/e2e/plugin-lifecycle-matrix-docker.sh, test/e2e/qa-lab/plugins/plugin-lifecycle-probe-runtime.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. labels Jul 3, 2026
@RomneyDa
RomneyDa force-pushed the dallin/wave1-docker-runtime branch from 4aefa4d to 37783c3 Compare July 3, 2026 23:52
@clawsweeper clawsweeper Bot removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 4, 2026
@RomneyDa RomneyDa mentioned this pull request Jul 4, 2026
Closed
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. labels Jul 4, 2026
@RomneyDa
RomneyDa merged commit ab9e233 into main Jul 4, 2026
204 of 212 checks passed
@RomneyDa
RomneyDa deleted the dallin/wave1-docker-runtime branch July 4, 2026 01:13
@clawsweeper clawsweeper Bot mentioned this pull request Jul 4, 2026
Merged
@github-actions github-actions Bot mentioned this pull request Jul 4, 2026
Open
@Haderach-Ram Haderach-Ram mentioned this pull request Jul 4, 2026
Open
@RomneyDa RomneyDa mentioned this pull request Jul 4, 2026
Merged
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 4, 2026
@RomneyDa
improve(qa): execute runtime scenarios through Docker (openclaw#99705)
5e5ef5a 
* test(qa): run Docker runtime scenarios end to end

* test(qa): keep plugin lifecycle probe runtime-only
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

extensions: qa-lab maintainer Maintainer-authored PR merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@RomneyDa