Skip to content

improve(qa): standardize script evidence output#99374

Merged
RomneyDa merged 1 commit into
mainfrom
dallin/qa-script-evidence-writer
Jul 3, 2026
Merged

improve(qa): standardize script evidence output#99374
RomneyDa merged 1 commit into
mainfrom
dallin/qa-script-evidence-writer

Conversation

@RomneyDa

@RomneyDa RomneyDa commented Jul 3, 2026

Copy link
Copy Markdown
Member

What Problem This Solves

Resolves a QA maintenance problem where script-backed scenarios assembled qa-evidence.json, logs, and artifact references independently. This allowed unbounded diagnostic output, inconsistent artifact paths, and blocked prerequisite messages to be lost after output truncation.

Why This Change Was Made

Adds one shared QA script evidence writer that owns schema construction, UTF-8 byte-bounded logs and failure details, contained artifact paths, and streaming blocked-status detection. The existing hosted-media and ClawHub release-candidate producers now use the shared path instead of maintaining separate evidence plumbing.

User Impact

There is no user-visible product behavior change. QA maintainers get consistent pass/fail/blocked evidence, reliable artifact links, and bounded diagnostics for script-backed scenarios.

Evidence

@clawsweeper

clawsweeper Bot commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 3, 2026, 1:19 AM ET / 05:19 UTC.

Summary
The PR adds a shared QA script evidence writer, exports the needed QA Lab evidence helpers through the QA Lab API barrel, and migrates the hosted-media and ClawHub release-candidate producers plus focused tests to the shared path.

PR surface: Source +9, Tests +371. Total +380 across 7 files.

Reproducibility: not applicable. This PR is internal QA maintenance, not a reported product bug. The source comparison does show two current-main script producers carrying duplicated evidence-writing behavior.

Review metrics: 2 noteworthy metrics.

  • QA Lab API exports: 7 added. The PR promotes evidence-summary helpers and types through the QA Lab API barrel, an additive API-surface change maintainers should consciously accept.
  • Script producers migrated: 2 changed. Hosted media and ClawHub release-candidate evidence now share one writer, so reviewer attention should focus on preserving artifact, status, and output-file semantics for both.

Stored data model
Persistent data-model change detected: serialized state: test/e2e/qa-lab/media/hosted-media-provider-live.ts, serialized state: test/e2e/qa-lab/plugins/clawhub-release-candidate-install.test.ts, serialized state: test/e2e/qa-lab/runtime/script-evidence.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🌊 off-meta tidepool
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • none.

Risk before merge

  • [P1] The PR body's linked Actions proof run is cancelled in live GitHub data; maintainers should use the current PR CI/Testbox evidence before merge.

Maintainer options:

  1. Decide the mitigation before merge
    Land the shared QA script evidence writer after current CI/proof is green, keeping the helper in the QA test/runtime layer and the QA Lab evidence helpers exposed through the extension API barrel for boundary-safe test use.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • No automated repair lane is needed: the protected maintainer PR has no narrow code defect from this review and should proceed through normal maintainer CI/proof handling.

Security
Cleared: The diff only changes QA/test TypeScript evidence plumbing and an additive QA Lab API barrel export; it does not add dependencies, workflows, lockfile changes, secret handling, or executable supply-chain sources.

Review details

Best possible solution:

Land the shared QA script evidence writer after current CI/proof is green, keeping the helper in the QA test/runtime layer and the QA Lab evidence helpers exposed through the extension API barrel for boundary-safe test use.

Do we have a high-confidence way to reproduce the issue?

Not applicable; this PR is internal QA maintenance, not a reported product bug. The source comparison does show two current-main script producers carrying duplicated evidence-writing behavior.

Is this the best way to solve the issue?

Yes; placing the shared writer under the QA e2e runtime keeps the fix at the duplicated producer layer, and routing QA Lab evidence helpers through extensions/qa-lab/api.ts matches the scoped extension boundary guidance.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 328fb70ba730.

Label changes

Label justifications:

  • P3: This is low-risk internal QA evidence cleanup with no intended user-visible product behavior change.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The external-contributor proof gate is not applicable because this is a maintainer-labeled member PR; the PR still includes targeted test and Testbox/CI evidence for maintainer merge review.
Evidence reviewed

PR surface:

Source +9, Tests +371. Total +380 across 7 files.

View PR surface stats
Area Files Added Removed Net
Source 1 9 0 +9
Tests 6 578 207 +371
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 7 587 207 +380

What I checked:

  • Root policy read and applied: The root AGENTS.md was read fully; its PR review, protected-label, extension-boundary, and QA validation guidance applies to this PR. (AGENTS.md:1, 328fb70ba730)
  • Scoped extension policy read: The extension guide says tests that need bundled plugin helpers should use the extension public barrel instead of deep-importing private extension internals, which supports the PR's QA Lab API barrel export approach. (extensions/AGENTS.md:1, 328fb70ba730)
  • Protected maintainer label: Live PR metadata shows the protected maintainer label on this open PR, so cleanup-close is not allowed by this review workflow. (d0c1bfcb5913)
  • Current hosted-media duplication: Current main manually owns log creation, artifact paths, evidence summary construction, and qa-evidence/latest-run writes inside the hosted media producer. (test/e2e/qa-lab/media/hosted-media-provider-live.ts:724, 328fb70ba730)
  • Current ClawHub duplication: Current main has a parallel hand-written evidence path for the ClawHub release-candidate producer, including log artifact normalization and qa-evidence/latest-run writes. (test/e2e/qa-lab/plugins/clawhub-release-candidate-install.ts:358, 328fb70ba730)
  • Shared writer implementation: The PR head adds a test/QA-layer writer that normalizes contained artifact paths, builds script evidence through the QA Lab evidence contract, writes the bounded log, and writes qa-evidence/latest-run outputs. (test/e2e/qa-lab/runtime/script-evidence.ts:116, d0c1bfcb5913)

Likely related people:

What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. labels Jul 3, 2026
This was referenced Jul 3, 2026
Closed
Closed
Merged
@RomneyDa
RomneyDa merged commit 5361e5a into main Jul 3, 2026
185 of 195 checks passed
@RomneyDa
RomneyDa deleted the dallin/qa-script-evidence-writer branch July 3, 2026 19:59
@github-actions github-actions Bot mentioned this pull request Jul 3, 2026
Open
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 4, 2026
@RomneyDa
test(qa): share script evidence writer (openclaw#99374)
74e9585
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

extensions: qa-lab maintainer Maintainer-authored PR P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: L status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@RomneyDa