Skip to content

test: add executable runtime fixture canaries#99737

Merged
RomneyDa merged 2 commits into
mainfrom
dallin/wave1-runtime-fixtures
Jul 4, 2026
Merged

test: add executable runtime fixture canaries#99737
RomneyDa merged 2 commits into
mainfrom
dallin/wave1-runtime-fixtures

Conversation

@RomneyDa

@RomneyDa RomneyDa commented Jul 4, 2026

Copy link
Copy Markdown
Member

What Problem This Solves

Several QA scenario rows described media delivery, browser Talk lifecycle, Voice Call entry points, heartbeat scheduling, and OpenTelemetry behavior without executing the corresponding runtime boundaries. That left the scenario catalog able to report coverage from helper-only or design-level checks rather than reusable executable fixtures.

Why This Change Was Made

This adds bounded canaries at the existing owner seams: mock speech synthesis followed by the real WebChat media-ticket route, a real Playwright page for browser Talk start/stop, mock realtime and Voice Call providers behind actual Gateway/plugin entry points, a real-timer heartbeat schedule check, and direct OTEL producer execution through the shared script evidence writer. The active-run control fixture remains secondary evidence because its embedded-run dependencies are intentionally injected test boundaries.

User Impact

There is no production behavior change. Maintainers get executable, reusable QA evidence for media, Talk, Voice Call, scheduler, and telemetry paths, with evidence claims constrained to the boundaries each fixture actually runs.

Evidence

  • node scripts/run-vitest.mjs test/e2e/qa-lab/media/webchat-auto-tts.e2e.test.ts test/e2e/qa-lab/voice/active-talk-agent-run-status.e2e.test.ts extensions/voice-call/src/voice-call-cli-rpc-agent-tool.e2e.test.ts src/infra/heartbeat-runner.active-hours-schedule.e2e.test.ts test/e2e/qa-lab/runtime/qa-otel-smoke.e2e.test.ts extensions/qa-lab/src/test-file-scenario-runner.test.ts extensions/qa-lab/src/scenario-catalog.test.ts — 98 assertions passed across the E2E and QA Lab shards.
  • node scripts/run-vitest.mjs run --config test/vitest/vitest.ui-e2e.config.ts --configLoader runner ui/src/ui/e2e/browser-talk-start-stop.e2e.test.ts --reporter=verbose — Playwright Talk lifecycle passed.
  • Direct qa-otel-smoke-runtime.ts execution — shared evidence status pass, relative summary artifact resolved correctly, and traces/metrics/correlated logs captured.
  • Testbox-through-Crabbox tbx_01kwnapfrp056k2ndxkmrmx2wrcheck:changed passed typecheck, lint, import-cycle, and repository guard lanes. Actions run: https://github.com/openclaw/openclaw/actions/runs/28690272139
  • Final autoreview: no accepted/actionable findings.

@openclaw-barnacle openclaw-barnacle Bot added channel: voice-call Channel integration: voice-call app: web-ui App: web-ui extensions: qa-lab size: L maintainer Maintainer-authored PR labels Jul 4, 2026
@clawsweeper

clawsweeper Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 3, 2026, 9:35 PM ET / 01:35 UTC.

Summary
Adds executable QA canaries and scenario metadata for Voice Call CLI/RPC/tool entry points, WebChat TTS media tickets, browser Talk start/stop, heartbeat scheduling, and OTEL script evidence.

PR surface: Source +44, Tests +672, Other +72. Total +788 across 14 files.

Reproducibility: not applicable. This PR adds executable QA coverage rather than fixing a reported runtime bug. The relevant verification path is the added canary tests and current-head CI/Testbox evidence.

Review metrics: none identified.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

  • [P1] Live PR checks still showed QA Smoke CI and two compact whole-suite jobs pending during review; merge should wait for required checks to finish.

Maintainer options:

  1. Decide the mitigation before merge
    Keep the PR open for maintainer review and merge once the current-head required checks finish, preserving the executable QA canaries in test/QA-owned surfaces.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • No ClawSweeper repair lane is needed because there are no concrete review findings; remaining action is maintainer review/merge after required CI completes.

Security
Cleared: The diff is test/QA-only and does not add dependency, workflow, lockfile, credential, or production code-execution surface concerns.

Review details

Best possible solution:

Keep the PR open for maintainer review and merge once the current-head required checks finish, preserving the executable QA canaries in test/QA-owned surfaces.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR adds executable QA coverage rather than fixing a reported runtime bug. The relevant verification path is the added canary tests and current-head CI/Testbox evidence.

Is this the best way to solve the issue?

Yes: the coverage is placed at existing owner seams for Voice Call, Gateway/WebChat media, browser Talk, heartbeat scheduling, and QA OTEL evidence without adding production APIs or config.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 0f1fbbe8779d.

Label changes

Label changes:

  • add rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The contributor proof gate does not apply to this maintainer-labeled PR; the PR body still lists targeted runtime tests, Playwright proof, direct OTEL producer execution, Testbox check:changed, and autoreview evidence.

Label justifications:

  • P3: This is low-risk test and QA evidence coverage with no production behavior change.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The contributor proof gate does not apply to this maintainer-labeled PR; the PR body still lists targeted runtime tests, Playwright proof, direct OTEL producer execution, Testbox check:changed, and autoreview evidence.
Evidence reviewed

PR surface:

Source +44, Tests +672, Other +72. Total +788 across 14 files.

View PR surface stats
Area Files Added Removed Net
Source 2 61 17 +44
Tests 6 696 24 +672
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 6 91 19 +72
Total 14 848 60 +788

What I checked:

Likely related people:

  • RomneyDa: Current-main history shows Dallin Romney recently touched the central QA/runtime, Voice Call, UI helper, and heartbeat files, and the PR head is in the same area. (role: recent area contributor; confidence: high; commits: e6dec69403f0, ab9e233d02a8, e1338bb2fa98; files: extensions/voice-call/src/manager.test-harness.ts, extensions/voice-call/index.ts, test/e2e/qa-lab/runtime/qa-otel-smoke-runtime.ts)
  • Wynne668: Recent current-main history includes a Control UI helper/chat-flow change adjacent to the browser E2E helper touched by this PR. (role: recent UI adjacent contributor; confidence: medium; commits: 61564f2d2e83; files: ui/src/test-helpers/control-ui-e2e.ts, ui/src/ui/e2e/chat-flow.e2e.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. labels Jul 4, 2026
@RomneyDa
RomneyDa force-pushed the dallin/wave1-runtime-fixtures branch from d671d2c to c807de8 Compare July 4, 2026 01:27
@clawsweeper clawsweeper Bot added rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 4, 2026
@RomneyDa
RomneyDa merged commit 3b4092d into main Jul 4, 2026
118 of 123 checks passed
@RomneyDa
RomneyDa deleted the dallin/wave1-runtime-fixtures branch July 4, 2026 03:22
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 4, 2026
* test(qa): add runtime fixture canaries

* test(voice-call): track fixture temp directories
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

app: web-ui App: web-ui channel: voice-call Channel integration: voice-call extensions: qa-lab maintainer Maintainer-authored PR P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. size: L status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant