Skip to content

fix(qa): stabilize primary smoke runtime evidence#99784

Merged
RomneyDa merged 3 commits into
mainfrom
qa-smoke-flake
Jul 4, 2026
Merged

fix(qa): stabilize primary smoke runtime evidence#99784
RomneyDa merged 3 commits into
mainfrom
qa-smoke-flake

Conversation

@RomneyDa

@RomneyDa RomneyDa commented Jul 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Related: #99705
Related: #99737
Related: #99368

What Problem This Solves

Fixes two recent QA Smoke CI regressions while preserving the stronger primary runtime coverage:

  1. Smoke-linked Docker scenarios each rebuilt the complete OpenClaw package, growing the lane from roughly 5–6 minutes to 11–13 minutes and sometimes exhausting the 8 GB Node heap.
  2. test: add executable runtime fixture canaries #99737 converted qa-otel-smoke to executable script evidence, but the profile runner expands ${outputDir} to an absolute path while the child qa suite CLI requires a repo-relative output path. That made current-main smoke runs fail after all gateway scenarios and both Docker scenarios had passed.

Why This Change Was Made

  • Prepare one validated candidate tarball before the profile and expose it through the existing OPENCLAW_CURRENT_PACKAGE_TGZ Docker E2E contract.
  • Keep all six release-category scenarios on Docker; taxonomy category IDs remain the single source of truth for profile selection.
  • Give the single package build a 12 GB Node heap to clear the observed 8 GB tsdown ceiling.
  • Translate the OTEL producer's outer artifact path into the repo-relative form required by its child QA command, while rejecting paths outside the repository.

This does not move primary evidence back to focused Vitest tests and does not add a profile-specific scenario allowlist.

User Impact

Contributors retain packaged-runtime Docker coverage without duplicate package builds. The OTEL runtime producer can also run correctly through the profile's shared script adapter instead of deterministically failing after the rest of the smoke profile succeeds.

Evidence

@clawsweeper

clawsweeper Bot commented Jul 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 3, 2026, 11:49 PM ET / 03:49 UTC.

Summary
This PR changes QA Smoke CI to build one reusable Docker package tarball, export OPENCLAW_CURRENT_PACKAGE_TGZ, raise the package-prep Node heap, and normalize the OTEL smoke child qa suite output directory to a repo-relative path.

PR surface: Tests +33, Config +5. Total +38 across 4 files.

Reproducibility: yes. source inspection plus PR CI context gives a high-confidence path: current main lacks a shared tarball export for smoke Docker lanes and passes profile-expanded absolute OTEL output dirs into a child qa suite path that expects repo-relative input. I did not run the full smoke profile locally.

Review metrics: 3 noteworthy metrics.

  • QA Smoke package preparation: 1 shared package prep added. The main automation behavior change is that package-aware smoke Docker lanes should reuse one validated tarball instead of each rebuilding the package.
  • CI package heap cap: 8 GB -> 12 GB. The required smoke lane now gives package preparation more Node heap headroom, which maintainers should notice before merge.
  • OTEL child output path: 1 child QA suite arg normalized. The OTEL producer now translates profile-expanded absolute output paths back to the repo-relative form expected by the child QA suite.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Let exact-head QA Smoke CI finish successfully or record an explicit maintainer proof override before merge.

Risk before merge

  • [P1] Exact-head QA Smoke CI was still in progress when checked, so the changed required smoke path has not completed live automation on 59b9f266348c490ac639cf720e0d127ed486fa16.
  • [P1] The workflow intentionally replaces the narrower qaRuntime prebuild with full package preparation and raises package-prep heap headroom from 8 GB to 12 GB, which is a CI runner-resource tradeoff maintainers should explicitly accept before merge.

Maintainer options:

  1. Wait for exact-head smoke proof (recommended)
    Let exact-head QA Smoke CI complete successfully on 59b9f266348c490ac639cf720e0d127ed486fa16 before merge because the PR changes that required workflow path.
  2. Accept targeted proof deliberately
    Maintainers can choose to merge before the exact-head smoke job finishes only if they explicitly accept the PR-body Testbox/Actions proof and the 12 GB heap tradeoff as sufficient.
  3. Pause if smoke still fails
    If exact-head QA Smoke keeps failing or timing out, keep this PR open and repair the shared-package workflow or runner resource settings before landing.

Next step before merge

  • [P2] The PR has no concrete code repair finding, but the protected maintainer label and in-progress exact-head QA Smoke proof make this a maintainer merge-timing decision.

Security
Cleared: The diff uses existing repository scripts inside an existing CI job and does not add third-party Actions, permissions, dependencies, lockfile changes, secrets exposure, or new external code execution sources.

Review details

Best possible solution:

Land after exact-head QA Smoke and required CI finish successfully, or with explicit maintainer acceptance of the targeted proof and runner-resource tradeoff.

Do we have a high-confidence way to reproduce the issue?

Yes, source inspection plus PR CI context gives a high-confidence path: current main lacks a shared tarball export for smoke Docker lanes and passes profile-expanded absolute OTEL output dirs into a child qa suite path that expects repo-relative input. I did not run the full smoke profile locally.

Is this the best way to solve the issue?

Yes, this is the right layer for preserving stronger Docker smoke coverage: reuse the existing OPENCLAW_CURRENT_PACKAGE_TGZ contract once in the workflow and localize the output-dir conversion to the OTEL producer. The remaining question is exact-head automation proof, not a better implementation location.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 9d68f877ac3e.

Label changes

Label changes:

  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🦞 diamond lobster.
  • remove rating: 🦐 gold shrimp: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.

Label justifications:

  • P2: This is a normal-priority QA Smoke CI regression fix with limited product-runtime blast radius but meaningful maintainer validation impact.
  • merge-risk: 🚨 automation: The PR changes required CI workflow behavior and Docker package preparation, so merge safety depends on exact-head automation proof beyond source correctness.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The external-contributor proof gate does not apply to this MEMBER-authored, maintainer-labeled PR; its PR body still includes Testbox, Actions, focused Vitest, formatting, and autoreview evidence for maintainers to evaluate.
Evidence reviewed

PR surface:

Tests +33, Config +5. Total +38 across 4 files.

View PR surface stats
Area Files Added Removed Net
Source 0 0 0 0
Tests 3 36 3 +33
Docs 0 0 0 0
Config 1 6 1 +5
Generated 0 0 0 0
Other 0 0 0 0
Total 4 42 4 +38

What I checked:

  • Repository policy read: Root AGENTS.md and scoped scripts, test, and extensions guides were read fully; protected-label, deep PR review, script-wrapper, test, and extension-boundary guidance affected the review. (AGENTS.md:1, 9d68f877ac3e)
  • Live PR state: Live GitHub reports head 59b9f266348c490ac639cf720e0d127ed486fa16, mergeable but unstable, with the protected maintainer label and merge-risk: 🚨 automation label present. (59b9f266348c)
  • Combined PR file list: The net PR diff changes only the CI workflow, OTEL smoke runtime/test, and workflow guard test; the transient scenario YAML changes in the first patch-series commit are not part of the combined PR state. (59b9f266348c)
  • Current main smoke workflow: Current main's QA Smoke CI path runs scripts/build-all.mjs qaRuntime with an 8 GB Node heap and does not export OPENCLAW_CURRENT_PACKAGE_TGZ before running the smoke profile. (.github/workflows/ci.yml:873, 9d68f877ac3e)
  • PR smoke workflow: The PR head prepares .artifacts/qa-e2e/smoke-ci-package/openclaw-current.tgz once with scripts/package-openclaw-for-docker.mjs, exports it through OPENCLAW_CURRENT_PACKAGE_TGZ, and raises the heap cap to 12 GB for package prep. (.github/workflows/ci.yml:873, 59b9f266348c)
  • Package reuse contract: docker_e2e_prepare_package_tgz already accepts an existing package tarball path or OPENCLAW_CURRENT_PACKAGE_TGZ, verifies the file exists, and returns its absolute path instead of repacking. (scripts/lib/docker-e2e-package.sh:159, 9d68f877ac3e)

Likely related people:

  • RomneyDa: Recent merged PRs changed QA scenario Docker routing, executable runtime fixtures, and smoke-profile behavior that this PR builds on; the PR author also appears in current-main history for the implicated QA surfaces. (role: recent QA Lab and smoke Docker contributor; confidence: high; commits: ab9e233d02a8, 3b4092dbaa96, 19035bdca1d9; files: qa/scenarios/runtime/qa-otel-smoke.yaml, test/e2e/qa-lab/runtime/qa-otel-smoke-runtime.ts, test/e2e/qa-lab/runtime/docker-e2e-lane.ts)
  • vincentkoc: Recent commits on the package-for-Docker and Docker package helper contracts make this a likely routing candidate for shared tarball reuse behavior. (role: Docker package helper adjacent contributor; confidence: high; commits: f53fdb688df8, 7cc0879d0e80, 851b65c060bb; files: scripts/package-openclaw-for-docker.mjs, scripts/lib/docker-e2e-package.sh)
  • Colin Johnson: Commit 4ae0a5d958a0 added the QA Smoke CI workflow path and guard tests that this PR now changes. (role: CI smoke profile introducer; confidence: medium; commits: 4ae0a5d958a0; files: .github/workflows/ci.yml, test/scripts/ci-workflow-guards.test.ts, docs/ci.md)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. labels Jul 4, 2026
@RomneyDa RomneyDa changed the title fix(qa): restore fast smoke profile evidence fix(qa): reuse one package across smoke Docker lanes Jul 4, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jul 4, 2026
@RomneyDa RomneyDa changed the title fix(qa): reuse one package across smoke Docker lanes fix(qa): stabilize primary smoke runtime evidence Jul 4, 2026
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Jul 4, 2026
@clawsweeper clawsweeper Bot mentioned this pull request Jul 4, 2026
Merged
@RomneyDa
RomneyDa merged commit 26cb189 into main Jul 4, 2026
94 of 100 checks passed
@RomneyDa
RomneyDa deleted the qa-smoke-flake branch July 4, 2026 03:59
This was referenced Jul 4, 2026
Closed
Merged
@github-actions github-actions Bot mentioned this pull request Jul 4, 2026
Open
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 4, 2026
@RomneyDa
fix(qa): stabilize primary smoke runtime evidence (openclaw#99784)
98c3b4c 
* fix(qa): restore fast smoke profile evidence

* fix(qa): reuse one package across smoke Docker lanes

* fix(qa): keep OTEL smoke output repo-relative
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

maintainer Maintainer-authored PR merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. P2 Normal backlog priority with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: XS status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@RomneyDa