Skip to content

Migrate Tool Search gateway E2E into QA Lab flow#97478

Merged
RomneyDa merged 7 commits into
mainfrom
dallin/qa-tool-search-flow-experiment
Jun 29, 2026
Merged

Migrate Tool Search gateway E2E into QA Lab flow#97478
RomneyDa merged 7 commits into
mainfrom
dallin/qa-tool-search-flow-experiment

Conversation

@RomneyDa

@RomneyDa RomneyDa commented Jun 28, 2026

Copy link
Copy Markdown
Member

What Problem This Solves

The Tool Search gateway E2E was still owned by a standalone script, so the QA scenario only linked out to script evidence instead of running as a native QA Lab flow.

This keeps the existing coverage IDs unchanged:

  • Primary: runtime.hosted-tool-use
  • Secondary: plugins.plugin-tools, tools.invocation

Why This Change Was Made

The standalone scripts/tool-search-gateway-e2e.ts path duplicated QA Lab lifecycle work. This moves the scenario into a flow-backed QA Lab fixture that stages the fake plugin catalog, restarts the QA gateway with scenario config, drives /v1/responses, and compares provider/debug/session-log evidence from inside the QA runner.

The mock provider ordering was also tightened so Tool Search follow-up summaries are returned before generic QA summary text.

User Impact

Maintainers can now run the Tool Search gateway proof through the normal QA Lab scenario entrypoint:

pnpm openclaw qa suite --provider-mode mock-openai --scenario tool-search-gateway-e2e

No taxonomy changes are included.

Evidence

  • node scripts/run-vitest.mjs run extensions/qa-lab/src/tool-search-gateway.fixture.test.ts extensions/qa-lab/src/providers/mock-openai/server.test.ts
  • pnpm openclaw qa suite --provider-mode mock-openai --scenario tool-search-gateway-e2e --output-dir .artifacts/qa-e2e/tool-search-gateway-flow-experiment
  • node scripts/run-vitest.mjs run extensions/qa-lab/src/scenario-catalog.test.ts extensions/qa-lab/src/scenario-packs.test.ts
  • pnpm format:docs:check -- docs/tools/tool-search.md
  • git diff --check
  • .agents/skills/autoreview/scripts/autoreview --mode local

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation scripts Repository scripts extensions: qa-lab size: XL maintainer Maintainer-authored PR labels Jun 28, 2026
@clawsweeper

clawsweeper Bot commented Jun 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 28, 2026, 9:12 PM ET / 01:12 UTC.

Summary
The PR migrates Tool Search gateway E2E coverage from scripts/tool-search-gateway-e2e.ts into a QA Lab flow fixture, updates the scenario/docs/test ownership, and adjusts mock OpenAI Tool Search summary ordering.

PR surface: Source +826, Tests +96, Docs 0, Other -653. Total +269 across 9 files.

Reproducibility: not applicable. as a QA migration PR rather than a bug report. Source inspection verifies the previous mock-provider selection finding is fixed on the latest head, and the PR body lists focused Vitest, QA suite, docs-format, diff-check, and autoreview evidence that I did not rerun in this read-only review.

Review metrics: 2 noteworthy metrics.

  • QA proof entrypoint: 1 script removed, 1 flow scenario added. The merge changes how maintainers run the Tool Search gateway proof, so the replacement command needs explicit owner acceptance.
  • Mock-only scenario gate: 1 requiredProviderMode added. The provider-mode gate keeps this mock-backed flow out of implicit live-provider QA suite runs.

Stored data model
Persistent data-model change detected: serialized state: extensions/qa-lab/src/providers/mock-openai/server.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: canonical
Canonical: #97478
Summary: This PR is the active follow-up for migrating the Tool Search gateway proof from a standalone script into a native QA Lab flow; related merged PRs are provenance, not replacements.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🌊 off-meta tidepool
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Confirm the QA Lab suite command is the supported replacement for the removed standalone script after focused QA evidence and required CI are green.

Risk before merge

  • [P1] Merging intentionally removes the standalone Tool Search gateway E2E script, so maintainers should accept the QA Lab suite command as the supported replacement proof path before landing.
  • [P1] This changes QA automation ownership rather than product runtime behavior; green unit tests alone do not prove downstream operator scripts no longer reference the deleted script.

Maintainer options:

  1. Accept the QA Lab replacement (recommended)
    Merge after maintainers explicitly accept that pnpm openclaw qa suite --provider-mode mock-openai --scenario tool-search-gateway-e2e replaces the removed standalone script and focused QA evidence is green.
  2. Keep a wrapper if operators still need it
    If a standalone command is still a supported maintainer workflow, keep or add a thin wrapper that delegates to the QA Lab scenario before removing the old script path.
  3. Pause if the proof path is unsettled
    If maintainers are not ready to retire the direct script entrypoint, pause this PR until the supported Tool Search proof contract is decided.

Next step before merge

  • [P2] No narrow automated repair remains; the maintainer-labeled PR needs human acceptance of the script-to-flow proof replacement and normal merge validation.

Security
Cleared: The diff changes internal QA fixtures, docs, scenario YAML, and mock-provider behavior without dependency, lockfile, workflow, permission, secret-handling, or external code-download changes.

Review details

Best possible solution:

Land the flow migration after maintainers accept the QA Lab suite command as the replacement proof path and the focused QA scenario validation is green.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a QA migration PR rather than a bug report. Source inspection verifies the previous mock-provider selection finding is fixed on the latest head, and the PR body lists focused Vitest, QA suite, docs-format, diff-check, and autoreview evidence that I did not rerun in this read-only review.

Is this the best way to solve the issue?

Yes, this is the best owner-boundary direction: the scenario catalog and QA Lab flow runner now own the proof instead of a standalone script. The remaining question is maintainer acceptance of retiring the direct script entrypoint, not a concrete patch defect.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 081f8fd1fa68.

Label changes

Label changes:

  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The external-contributor real-proof gate does not apply to this MEMBER-authored, maintainer-labeled QA automation PR; the PR body still includes terminal QA suite and focused test evidence for maintainer review.
  • remove rating: 🧂 unranked krab: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.
  • remove status: ⏳ waiting on author: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

  • P3: This is maintainer QA automation cleanup with no user-facing runtime behavior change.
  • merge-risk: 🚨 automation: The diff deletes a standalone E2E script and replaces it with QA Lab flow automation, changing the proof entrypoint maintainers may rely on.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The external-contributor real-proof gate does not apply to this MEMBER-authored, maintainer-labeled QA automation PR; the PR body still includes terminal QA suite and focused test evidence for maintainer review.
Evidence reviewed

PR surface:

Source +826, Tests +96, Docs 0, Other -653. Total +269 across 9 files.

View PR surface stats
Area Files Added Removed Net
Source 4 836 10 +826
Tests 2 117 21 +96
Docs 1 2 2 0
Config 0 0 0 0
Generated 0 0 0 0
Other 2 68 721 -653
Total 9 1023 754 +269

What I checked:

  • Repository policy read: Root, docs, extensions, and scripts AGENTS.md files were read; their review-depth, docs, extension-boundary, scripts, protected-label, and QA validation guidance shaped this review. (AGENTS.md:1, 081f8fd1fa68)
  • Live PR state: Live GitHub state shows the PR is open, maintainer-labeled, mergeable, and currently at head c5b42aa, which is newer than the compact context head. (c5b42aa2f995)
  • Current main remains script-backed: Current main still records tool-search-gateway-e2e as execution.kind: script with path scripts/tool-search-gateway-e2e.ts, so the central flow migration is not already implemented on main. (qa/scenarios/runtime/tool-search-gateway-e2e.yaml:25, 081f8fd1fa68)
  • PR head is flow-backed and mock-gated: The latest PR head changes the scenario to execution.kind: flow, adds requiredProviderMode: mock-openai, and asserts the runtime provider mode before staging the fake plugin fixture. (qa/scenarios/runtime/tool-search-gateway-e2e.yaml:25, c5b42aa2f995)
  • Suite planner honors provider gates: Implicit QA suite selection filters flow scenarios by execution.config.requiredProviderMode, which addresses the previous risk of selecting a mock-only scenario in the default live-provider lane. (extensions/qa-lab/src/suite-planning.ts:40, c5b42aa2f995)
  • Fixture requires the mock provider and validates both lanes: The new fixture asserts a mock provider URL, restarts the QA gateway after lane config mutation, drives /v1/responses, and asserts direct and compact Tool Search lane behavior including session-log mentions and payload reduction. (extensions/qa-lab/src/tool-search-gateway.fixture.ts:363, c5b42aa2f995)

Likely related people:

  • RomneyDa: Authored the merged predecessor that linked the Tool Search gateway proof into QA Lab inventory and authored this follow-up flow migration on the same QA scenario surface. (role: recent area contributor; confidence: high; commits: 39c4c02653ea, 1061882269db, c5b42aa2f995; files: qa/scenarios/runtime/tool-search-gateway-e2e.yaml, extensions/qa-lab/src/tool-search-gateway.fixture.ts, extensions/qa-lab/src/fixture-utils.ts)
  • steipete: Authored the merged core Tool Search PR that introduced the runtime feature, docs, mock-provider support, and original gateway E2E proof path this PR migrates. (role: feature introducer; confidence: high; commits: 15b26be8daf0, b85cb4e8d13e, b78604a0774f; files: src/agents/tool-search.ts, docs/tools/tool-search.md, scripts/tool-search-gateway-e2e.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. labels Jun 28, 2026
Base automatically changed from dallin/qa-tool-search-gateway-coverage to main June 28, 2026 11:28
@RomneyDa
RomneyDa force-pushed the dallin/qa-tool-search-flow-experiment branch from 193663f to fe78b94 Compare June 28, 2026 11:38
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 28, 2026
@RomneyDa
RomneyDa force-pushed the dallin/qa-tool-search-flow-experiment branch from 39475f0 to 815883e Compare June 29, 2026 00:03
@RomneyDa
RomneyDa force-pushed the dallin/qa-tool-search-flow-experiment branch from 393cb5c to df8abb7 Compare June 29, 2026 00:54
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jun 29, 2026
@clawsweeper clawsweeper Bot added status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 29, 2026
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 29, 2026
@RomneyDa
RomneyDa merged commit 4b8a0a8 into main Jun 29, 2026
104 of 108 checks passed
@RomneyDa
RomneyDa deleted the dallin/qa-tool-search-flow-experiment branch June 29, 2026 01:16
@clawsweeper clawsweeper Bot mentioned this pull request Jun 29, 2026
Merged
@github-actions github-actions Bot mentioned this pull request Jun 29, 2026
Open
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 29, 2026
@RomneyDa
Migrate Tool Search gateway E2E into QA Lab flow (openclaw#97478)
e3ac1cf 
* test: migrate tool search gateway to QA flow

* test: use built SDK in tool search QA fixture

* test: fix tool search QA fixture lint

* test: gate tool search QA flow to mock provider

* Share QA Lab fixture utilities

* Tighten QA fixture helper defaults

* test: gate tool search QA flow mode
QiuYuang pushed a commit to QiuYuang/openclaw that referenced this pull request Jul 1, 2026
@RomneyDa @QiuYuang
Migrate Tool Search gateway E2E into QA Lab flow (openclaw#97478)
00647f1 
* test: migrate tool search gateway to QA flow

* test: use built SDK in tool search QA fixture

* test: fix tool search QA fixture lint

* test: gate tool search QA flow to mock provider

* Share QA Lab fixture utilities

* Tighten QA fixture helper defaults

* test: gate tool search QA flow mode
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 1, 2026
@RomneyDa chenyangjun-xy
Migrate Tool Search gateway E2E into QA Lab flow (openclaw#97478)
4106750 
* test: migrate tool search gateway to QA flow

* test: use built SDK in tool search QA fixture

* test: fix tool search QA fixture lint

* test: gate tool search QA flow to mock provider

* Share QA Lab fixture utilities

* Tighten QA fixture helper defaults

* test: gate tool search QA flow mode
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 3, 2026
@RomneyDa @chenyangjun-xy
Migrate Tool Search gateway E2E into QA Lab flow (openclaw#97478)
d5267a6 
* test: migrate tool search gateway to QA flow

* test: use built SDK in tool search QA fixture

* test: fix tool search QA fixture lint

* test: gate tool search QA flow to mock provider

* Share QA Lab fixture utilities

* Tighten QA fixture helper defaults

* test: gate tool search QA flow mode
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

docs Improvements or additions to documentation extensions: qa-lab maintainer Maintainer-authored PR merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. scripts Repository scripts size: XL status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@RomneyDa