Skip to content

chore: add hosted media provider QA live evidence#97900

Merged
RomneyDa merged 1 commit into
mainfrom
dallin/qa-hosted-media-provider-live-tests
Jun 29, 2026
Merged

chore: add hosted media provider QA live evidence#97900
RomneyDa merged 1 commit into
mainfrom
dallin/qa-hosted-media-provider-live-tests

Conversation

@RomneyDa

@RomneyDa RomneyDa commented Jun 29, 2026

Copy link
Copy Markdown
Member

What Problem This Solves

The hosted media live provider tests already existed, but they were not linked into QA maturity evidence. Direct Vitest scenario wrappers would also overcount coverage when live tests skip because credentials are absent.

Why This Change Was Made

This adds QA Lab script scenarios for hosted image and video provider live coverage, and moves the live media CLI harness itself into test/e2e/qa-lab/media/hosted-media-provider-live.ts. The test:live:media:* package scripts now point at that QA Lab file directly, so there is one canonical live-media runner instead of a duplicated scripts/test-live-media.ts wrapper.

The QA evidence mode emits qa-evidence.json and reports missing live provider auth as blocked evidence instead of false passing primary coverage.

Coverage roles are intentionally narrow:

  • Primary: hosted-providers.image-generation-providers from the hosted image provider sweep through provider.generateImage.
  • Primary: hosted-providers.video-generation-providers from the hosted video provider sweep through provider.generateVideo.
  • Primary: media.reference-image-video-and-audio-inputs only on the video scenario, with full video modes forced so image/video reference paths are exercised where provider capabilities allow.
  • Secondary: media.video-generation-tool-invocation, because the existing live provider test backs the provider side but does not invoke the agent-facing video_generate tool.

User Impact

Release and QA operators can now include the hosted image/video provider live lanes in QA maturity evidence without duplicating live provider assertions or creating green evidence when credentials are missing. Existing pnpm test:live:media, pnpm test:live:media:image, pnpm test:live:media:music, and pnpm test:live:media:video commands continue to work through the QA Lab owner file.

Evidence

  • node scripts/run-vitest.mjs test/e2e/qa-lab/media/hosted-media-provider-live.test.ts test/scripts/test-projects.test.ts --reporter=verbose passed, 202 tests.
  • node scripts/run-node.mjs qa coverage --match hosted --json resolved both new script-backed scenarios and their coverage IDs through test/e2e/qa-lab/media/hosted-media-provider-live.ts.
  • git diff --check passed.
  • .agents/skills/autoreview/scripts/autoreview --mode local passed with no accepted/actionable findings after the final migration.

Live hosted media generation was not run locally for this PR; the new scenarios intentionally require live credentials and will report blocked evidence when none are available.

@clawsweeper

clawsweeper Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 29, 2026, 3:52 PM ET / 19:52 UTC.

Summary
The PR retargets the existing live media package scripts to a QA Lab hosted-media runner, adds hosted image/video script scenarios, and adds tests for the new evidence producer behavior.

PR surface: Tests +391, Config 0, Other +80. Total +471 across 8 files.

Reproducibility: not applicable. this is QA evidence wiring, not a bug report. The relevant path was traced from the new YAML scenarios through the QA Lab script runner and the hosted media producer.

Review metrics: 1 noteworthy metric.

  • QA evidence coverage: 2 scenarios added; 3 primary coverage IDs and 1 secondary coverage ID mapped. The PR changes which hosted media maturity targets can be backed by live QA evidence.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🌊 off-meta tidepool
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Run hosted image and video QA scenarios with live credentials before relying on them as maturity pass evidence.
  • [P2] Rerun or clear the Windows checkout CI failure before merge.

Risk before merge

  • [P1] Credentialed hosted image/video generation was not run on this PR, so the new scenarios should not be counted as passing maturity evidence until a live provider lane produces pass entries.
  • [P1] Required merge checks still need to be green or explicitly handled; the observed Windows failure happened during checkout and does not prove a code defect.

Maintainer options:

  1. Merge As Blocked-Aware Wiring (recommended)
    After required checks are green, maintainers can land the scenario and producer plumbing while keeping maturity promotion gated on a future credentialed hosted media pass.
  2. Wait For Live Provider Output
    Hold the PR until hosted image and video lanes run with credentials and attach qa-evidence.json or logs that show pass entries.
  3. Pause If QA Policy Requires Pass Proof
    If maturity-linked scenarios must not land before live pass evidence exists, pause this PR rather than landing blocked-only evidence wiring.

Next step before merge

  • [P2] The PR is protected by maintainer ownership and has no narrow automated repair finding; the next action is maintainer review, required CI completion, and a decision on whether blocked-aware QA wiring can land before credentialed live proof.

Security
Cleared: The diff adds repo-local QA scenarios/tests and retargets package scripts; it adds no dependency, workflow permission, secret handling, or third-party code execution surface beyond existing repo-local live-test execution.

Review details

Best possible solution:

Keep the QA Lab producer wiring and command compatibility, but require credentialed hosted image/video evidence before treating the new scenarios as maturity pass proof.

Do we have a high-confidence way to reproduce the issue?

Not applicable; this is QA evidence wiring, not a bug report. The relevant path was traced from the new YAML scenarios through the QA Lab script runner and the hosted media producer.

Is this the best way to solve the issue?

Yes; moving the existing hosted media harness into a QA Lab script producer is the right owner boundary and preserves the existing package script surface. The remaining caveat is operational proof, not a code-shape blocker.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against d43366b4e392.

Label changes

Label justifications:

  • P3: This is maintainer-facing QA evidence wiring with limited runtime blast radius and no user-facing product behavior change.
  • merge-risk: 🚨 automation: The diff changes QA proof capture and maturity-evidence automation, while credentialed hosted image/video output has not been produced on this PR.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The external contributor proof gate does not apply to this maintainer-labeled PR; the body includes local wiring validation and explicitly notes that live hosted media generation was not run.
Evidence reviewed

PR surface:

Tests +391, Config 0, Other +80. Total +471 across 8 files.

View PR surface stats
Area Files Added Removed Net
Source 0 0 0 0
Tests 4 655 264 +391
Docs 0 0 0 0
Config 1 4 4 0
Generated 0 0 0 0
Other 3 81 1 +80
Total 8 740 269 +471

What I checked:

  • Repository policy read: Root AGENTS.md and the scoped scripts/test guides were read fully; the maintainer-label protected-item rule and script-wrapper guidance apply to this review. (AGENTS.md:24, d43366b4e392)
  • Live PR state: Live GitHub metadata shows author RomneyDa, the maintainer label, head 2660d46, MERGEABLE/UNSTABLE state, and no human review comments. (2660d469f106)
  • Current main lacks this wiring: Current main still points test:live:media scripts at scripts/test-live-media.ts and has no hosted-media-provider-live runner or hosted image/video scenario IDs. (package.json:1883, d43366b4e392)
  • Package command compatibility: The PR preserves the four existing test:live:media script names while retargeting them to the QA Lab hosted-media runner path. (package.json:1883, 2660d469f106)
  • Scenario evidence mapping: The added image and video scenarios are script-backed live-only scenarios; the video scenario maps two primary coverage IDs and one secondary coverage ID, and both scenarios allow blocked evidence for missing credentials. (qa/scenarios/media/hosted-video-generation-providers-live.yaml:8, 2660d469f106)
  • Producer behavior: The new producer classifies no-auth/no-runnable-provider output as blocked, writes qa-evidence.json plus latest-run.json, and returns nonzero only for fail status. (test/e2e/qa-lab/media/hosted-media-provider-live.ts:710, 2660d469f106)

Likely related people:

  • RomneyDa: Authored this PR and recently merged the ClawHub release-candidate QA script producer that uses the same QA Lab producer-evidence pattern. (role: recent adjacent owner; confidence: high; commits: 2660d469f106, 80a3c515fd13; files: test/e2e/qa-lab/media/hosted-media-provider-live.ts, qa/scenarios/media/hosted-video-generation-providers-live.yaml, test/e2e/qa-lab/plugins/clawhub-release-candidate-install.ts)
  • vincentkoc: Current-main blame and PR metadata tie the existing live media harness and QA Lab script-runner support to recent merged work by this contributor. (role: current-main area contributor; confidence: medium; commits: c0ee7a1d20f0; files: scripts/test-live-media.ts, extensions/qa-lab/src/test-file-scenario-runner.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. labels Jun 29, 2026
@RomneyDa
RomneyDa force-pushed the dallin/qa-hosted-media-provider-live-tests branch from 081cd65 to 2660d46 Compare June 29, 2026 19:39
@openclaw-barnacle openclaw-barnacle Bot added scripts Repository scripts size: XL and removed size: L labels Jun 29, 2026
@clawsweeper clawsweeper Bot added the merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. label Jun 29, 2026
@RomneyDa
RomneyDa merged commit 05d8312 into main Jun 29, 2026
178 of 186 checks passed
@RomneyDa
RomneyDa deleted the dallin/qa-hosted-media-provider-live-tests branch June 29, 2026 20:01
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 30, 2026
QiuYuang pushed a commit to QiuYuang/openclaw that referenced this pull request Jul 1, 2026
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 1, 2026
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extensions: qa-lab maintainer Maintainer-authored PR merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. scripts Repository scripts size: XL status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant