Skip to content

test(qa): exercise Gateway and MCP scenarios over real transports#99735

Merged
RomneyDa merged 1 commit into
mainfrom
dallin/wave1-gateway-mcp
Jul 4, 2026
Merged

test(qa): exercise Gateway and MCP scenarios over real transports#99735
RomneyDa merged 1 commit into
mainfrom
dallin/wave1-gateway-mcp

Conversation

@RomneyDa

@RomneyDa RomneyDa commented Jul 4, 2026

Copy link
Copy Markdown
Member

What Problem This Solves

The gateway-smoke, mcp-gateway-connect-startup-retry, and mcp-plugin-tools-call QA scenarios claimed Gateway/MCP behavior from helper-level or in-memory tests instead of exercising the real process and transport boundaries.

Why This Change Was Made

  • Reuses the existing QA Gateway child fixture and adds an onListening lifecycle hook so a client can start after the real socket binds but before Gateway readiness.
  • Runs gateway-smoke against a real Gateway child and its real WebSocket/health surfaces.
  • Starts a real MCP stdio client before Gateway readiness. A fixture-plugin startup gate is released only after a recording proxy observes the real connect frame, deterministically exercising the production startup-unavailable retry response and protocol negotiation.
  • Registers a fixture plugin, starts the real plugin-tools stdio server, lists its tools, and calls memory_search with the MCP SDK client.
  • Converts all three scenario definitions to canonical script evidence with primary coverage.

User Impact

No product behavior changes. QA evidence now fails when the real Gateway or MCP process, readiness, retry, negotiation, plugin discovery, stdio transport, or tool invocation boundary regresses.

Evidence

  • Real scenario producers:
    • gateway-smoke: pass
    • mcp-plugin-tools-call: pass
    • mcp-gateway-connect-startup-retry: three consecutive frozen-base runs passed; each observed 2 retryable startup-unavailable responses, 3 connect frames, protocol 4, and MCP start 7.0–9.7 seconds before Gateway readiness.
  • Focused tests: node scripts/run-vitest.mjs extensions/qa-lab/src/gateway-child.test.ts extensions/qa-lab/src/scenario-catalog.test.ts test/e2e/qa-lab/runtime/gateway-smoke.e2e.test.ts src/mcp/plugin-tools-mcp-client.test.ts — 100 tests passed.
  • Focused lint: node scripts/run-oxlint.mjs --tsconfig tsconfig.json extensions/qa-lab/api.ts extensions/qa-lab/src/bundled-plugin-staging.ts extensions/qa-lab/src/gateway-child.ts extensions/qa-lab/src/gateway-child.test.ts test/e2e/qa-lab/runtime/gateway-mcp-real-transports.ts — passed.
  • Remote changed gate: delegated Blacksmith Testbox tbx_01kwnaep24a90x84dwp3dmghaw, check:changed exit 0, Actions run 28690164618.
  • Autoreview: clean; no accepted/actionable findings.

@clawsweeper

clawsweeper Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 3, 2026, 9:40 PM ET / 01:40 UTC.

Summary
The PR adds a QA Lab real-transport producer and converts three Gateway/MCP scenarios from Vitest-backed evidence to script-backed primary coverage.

PR surface: Source +85, Tests +723, Other +17. Total +825 across 8 files.

Reproducibility: not applicable. This PR is a QA coverage/evidence change, not a reported product bug. The PR body and current-head proof check provide real scenario producer evidence instead of a bug reproduction path.

Review metrics: 1 noteworthy metric.

  • Scenario Evidence Mode: 3 changed from vitest to script. The PR changes the QA automation contract for three Gateway/MCP scenarios, which is the main merge-risk surface maintainers need to notice.

Stored data model
Persistent data-model change detected: serialized state: extensions/qa-lab/src/gateway-child.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

  • [P1] Merging this promotes three QA scenarios from lighter helper/Vitest evidence to real process, WebSocket, and MCP stdio script evidence, so maintainers should accept the extra automation runtime and flake surface before landing.

Maintainer options:

  1. Land With Real-Transport Evidence (recommended)
    Keep the three scenarios as primary script evidence and accept the added child-process/socket/MCP automation cost after current-head checks finish green.
  2. Demote If CI Flakes
    If the current head shows repeat instability, keep the producer but move it to a narrower or soak-style lane instead of making all three scenarios primary script evidence.

Next step before merge

  • [P2] Protected maintainer-authored PR with an automation merge-risk decision; no narrow ClawSweeper repair is indicated.

Security
Cleared: The diff adds QA-only process/socket/MCP test orchestration and does not change workflows, dependencies, lockfiles, secrets handling, or product security boundaries.

Review details

Best possible solution:

Land the real-transport QA evidence path once maintainers accept the automation cost and current-head required checks remain green.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR is a QA coverage/evidence change, not a reported product bug. The PR body and current-head proof check provide real scenario producer evidence instead of a bug reproduction path.

Is this the best way to solve the issue?

Yes: using the existing QA Gateway child fixture, script evidence writer, and MCP/Gateway transport surfaces is the right boundary for proving these scenarios without changing product runtime behavior.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against fd0355d5b1a1.

Label changes

Label changes:

  • add merge-risk: 🚨 automation: The diff changes QA scenario execution to real child-process, WebSocket, and MCP stdio script automation that can affect CI/proof reliability after merge.
  • add proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes terminal-style real scenario producer results and the current-head Real behavior proof check completed successfully.
  • add rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes terminal-style real scenario producer results and the current-head Real behavior proof check completed successfully.

Label justifications:

  • P3: This is a low-risk maintainer QA evidence improvement with no intended product behavior change.
  • merge-risk: 🚨 automation: The diff changes QA scenario execution to real child-process, WebSocket, and MCP stdio script automation that can affect CI/proof reliability after merge.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes terminal-style real scenario producer results and the current-head Real behavior proof check completed successfully.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes terminal-style real scenario producer results and the current-head Real behavior proof check completed successfully.
Evidence reviewed

PR surface:

Source +85, Tests +723, Other +17. Total +825 across 8 files.

View PR surface stats
Area Files Added Removed Net
Source 3 92 7 +85
Tests 2 724 1 +723
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 3 34 17 +17
Total 8 850 25 +825

What I checked:

Likely related people:

  • steipete: Peter Steinberger has the largest path-history footprint across the QA/MCP files sampled and introduced the repo-backed QA suite runner in commit 4780788. (role: feature-history owner; confidence: high; commits: 4780788bbbf9, fcee2683736a, 610407730d42; files: extensions/qa-lab/src/gateway-child.ts, qa/scenarios/runtime/gateway-smoke.yaml, src/mcp/plugin-tools-serve.ts)
  • RomneyDa: Dallin Romney authored this PR and has recent merged/open QA transport work, including prior Gateway/MCP scenario coverage and QA smoke reliability work found in live PR search. (role: recent QA transport contributor; confidence: medium; commits: 02d017f5cd8a; files: test/e2e/qa-lab/runtime/gateway-mcp-real-transports.ts, extensions/qa-lab/src/gateway-child.ts, qa/scenarios/runtime/mcp-gateway-connect-startup-retry.yaml)
  • vincentkoc: Vincent Koc appears repeatedly in MCP/plugin-tools path history and current MCP plugin-tools PR search results, including commits around runtime seams and release QA stabilization. (role: MCP adjacent owner; confidence: medium; commits: 0f7d9c957093, 74e7b8d47b18, e085fa1a3ffd; files: src/mcp/channel-bridge.ts, src/mcp/plugin-tools-serve.ts, qa/scenarios/plugins/mcp-plugin-tools-call.yaml)
  • gumadeiras: Gustavo Madeira Santana has multiple QA Lab staging and scenario organization commits in the touched QA helper area. (role: QA staging contributor; confidence: medium; commits: 82fe6f50efc4, bb7e9823a886, d5933af80bf1; files: extensions/qa-lab/src/bundled-plugin-staging.ts, extensions/qa-lab/src/gateway-child.ts, qa/scenarios/runtime/gateway-smoke.yaml)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. labels Jul 4, 2026
@RomneyDa
RomneyDa force-pushed the dallin/wave1-gateway-mcp branch from 861c355 to 3e14214 Compare July 4, 2026 01:17
@RomneyDa

RomneyDa commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@RomneyDa
RomneyDa force-pushed the dallin/wave1-gateway-mcp branch from 3e14214 to e4a9694 Compare July 4, 2026 01:25
@RomneyDa
RomneyDa force-pushed the dallin/wave1-gateway-mcp branch from e4a9694 to 02d017f Compare July 4, 2026 01:31
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. and removed proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 4, 2026
@RomneyDa
RomneyDa merged commit 9d68f87 into main Jul 4, 2026
132 of 140 checks passed
@RomneyDa
RomneyDa deleted the dallin/wave1-gateway-mcp branch July 4, 2026 03:26
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extensions: qa-lab maintainer Maintainer-authored PR merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. size: L status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant