chore: add hosted media provider QA live evidence#97900
Conversation
|
Codex review: needs maintainer review before merge. Reviewed June 29, 2026, 3:52 PM ET / 19:52 UTC. Summary PR surface: Tests +391, Config 0, Other +80. Total +471 across 8 files. Reproducibility: not applicable. this is QA evidence wiring, not a bug report. The relevant path was traced from the new YAML scenarios through the QA Lab script runner and the hosted media producer. Review metrics: 1 noteworthy metric.
Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Security Review detailsBest possible solution: Keep the QA Lab producer wiring and command compatibility, but require credentialed hosted image/video evidence before treating the new scenarios as maturity pass proof. Do we have a high-confidence way to reproduce the issue? Not applicable; this is QA evidence wiring, not a bug report. The relevant path was traced from the new YAML scenarios through the QA Lab script runner and the hosted media producer. Is this the best way to solve the issue? Yes; moving the existing hosted media harness into a QA Lab script producer is the right owner boundary and preserves the existing package script surface. The remaining caveat is operational proof, not a code-shape blocker. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against d43366b4e392. Label changesLabel justifications:
Evidence reviewedPR surface: Tests +391, Config 0, Other +80. Total +471 across 8 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
081cd65 to
2660d46
Compare
What Problem This Solves
The hosted media live provider tests already existed, but they were not linked into QA maturity evidence. Direct Vitest scenario wrappers would also overcount coverage when live tests skip because credentials are absent.
Why This Change Was Made
This adds QA Lab script scenarios for hosted image and video provider live coverage, and moves the live media CLI harness itself into
test/e2e/qa-lab/media/hosted-media-provider-live.ts. Thetest:live:media:*package scripts now point at that QA Lab file directly, so there is one canonical live-media runner instead of a duplicatedscripts/test-live-media.tswrapper.The QA evidence mode emits
qa-evidence.jsonand reports missing live provider auth as blocked evidence instead of false passing primary coverage.Coverage roles are intentionally narrow:
hosted-providers.image-generation-providersfrom the hosted image provider sweep throughprovider.generateImage.hosted-providers.video-generation-providersfrom the hosted video provider sweep throughprovider.generateVideo.media.reference-image-video-and-audio-inputsonly on the video scenario, with full video modes forced so image/video reference paths are exercised where provider capabilities allow.media.video-generation-tool-invocation, because the existing live provider test backs the provider side but does not invoke the agent-facingvideo_generatetool.User Impact
Release and QA operators can now include the hosted image/video provider live lanes in QA maturity evidence without duplicating live provider assertions or creating green evidence when credentials are missing. Existing
pnpm test:live:media,pnpm test:live:media:image,pnpm test:live:media:music, andpnpm test:live:media:videocommands continue to work through the QA Lab owner file.Evidence
node scripts/run-vitest.mjs test/e2e/qa-lab/media/hosted-media-provider-live.test.ts test/scripts/test-projects.test.ts --reporter=verbosepassed, 202 tests.node scripts/run-node.mjs qa coverage --match hosted --jsonresolved both new script-backed scenarios and their coverage IDs throughtest/e2e/qa-lab/media/hosted-media-provider-live.ts.git diff --checkpassed..agents/skills/autoreview/scripts/autoreview --mode localpassed with no accepted/actionable findings after the final migration.Live hosted media generation was not run locally for this PR; the new scenarios intentionally require live credentials and will report blocked evidence when none are available.