Skip to content

test: link Tool Search gateway QA coverage#97374

Merged
RomneyDa merged 1 commit into
mainfrom
dallin/qa-tool-search-gateway-coverage
Jun 28, 2026
Merged

test: link Tool Search gateway QA coverage#97374
RomneyDa merged 1 commit into
mainfrom
dallin/qa-tool-search-gateway-coverage

Conversation

@RomneyDa

Copy link
Copy Markdown
Member

What Problem This Solves

Batch 2 of the QA maturity linking work had an existing Tool Search gateway E2E script, but no QA Lab scenario wrapper. That meant the evidence runner could not count this existing proof in the scenario inventory.

This PR does not change taxonomy and does not add new product behavior.

Why This Change Was Made

This adds one native QA scenario wrapper for scripts/tool-search-gateway-e2e.ts.

Primary coverage claimed:

Scenario Existing proof Primary coverage
tool-search-gateway-e2e scripts/tool-search-gateway-e2e.ts runtime.hosted-tool-use

Why primary: the script starts the QA mock OpenAI provider and a real Gateway, creates a temporary fake plugin with a large tool catalog, calls /v1/responses, and asserts both direct mode and Tool Search code mode invoke the selected plugin tool. It also verifies compact mode exposes only the tool_search_code bridge to the provider while still producing the plugin tool result.

Secondary coverage kept:

Coverage ID Why secondary only
plugins.plugin-tools The script stages a fake plugin and invokes its registered tool, but the scenario is about Tool Search gateway behavior rather than broad plugin-tool registration coverage.
tools.invocation The selected plugin tool is invoked through OpenClaw, but this scenario does not own the whole generic tool invocation contract.

Intentionally not claimed: gateway.tool-and-skill-apis. I checked it, but this script does not call the commands.list, tools.*, or skills.* Gateway RPC surface described by that taxonomy ID.

User Impact

The QA scenario inventory can now run and score the existing Tool Search gateway E2E proof through QA Lab. Operators get evidence for existing hosted tool-use behavior without changing runtime code.

Follow-up note: fully deleting scripts/tool-search-gateway-e2e.ts is plausible, but not in this PR. Today the QA flow DSL can run gateway calls and assertions, but it does not have a clean first-class primitive for staging a temporary plugin directory before gateway startup. The smallest durable migration would be a reusable QA Lab fixture/helper for temporary plugin tool catalogs, then a flow scenario can replace the standalone script.

Evidence

  • node scripts/run-vitest.mjs test/scripts/tool-search-gateway-e2e.test.ts extensions/qa-lab/src/scenario-catalog.test.ts extensions/qa-lab/src/coverage-report.test.ts extensions/qa-lab/src/scorecard-evidence.test.ts
    • 1 script helper test file passed, 11 tests
    • 3 QA extension test files passed, 47 tests
  • pnpm openclaw qa suite --provider-mode mock-openai --scenario tool-search-gateway-e2e --output-dir .artifacts/qa-e2e/tool-search-gateway-coverage
    • 1 scenario passed
    • evidence artifact: .artifacts/qa-e2e/tool-search-gateway-coverage/qa-evidence.json
  • git diff --check -- qa/scenarios/runtime/tool-search-gateway-e2e.yaml
  • .agents/skills/autoreview/scripts/autoreview --mode local
    • first run caught an overclaim for gateway.tool-and-skill-apis; accepted and removed
    • final run clean: no accepted/actionable findings

@clawsweeper

clawsweeper Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 28, 2026, 1:41 AM ET / 05:41 UTC.

Summary
Adds a QA Lab runtime scenario YAML wrapper for the existing Tool Search gateway E2E script, with primary hosted tool-use coverage and secondary plugin/tool invocation coverage.

PR surface: Other +28. Total +28 across 1 file.

Reproducibility: not applicable. this is a QA coverage wrapper rather than a bug report. Source review confirms the existing script and docs define the behavior being linked into QA Lab.

Review metrics: none identified.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • none.

Next step before merge

  • No automated repair is needed; this maintainer-labeled PR should continue through normal maintainer review and required checks.

Security
Cleared: The diff adds one QA scenario YAML file and does not alter executable code, dependencies, workflows, secrets, or package resolution.

Review details

Best possible solution:

Merge the focused scenario wrapper after normal maintainer review and required checks, leaving the broader temporary-plugin fixture migration as a separate QA Lab follow-up.

Do we have a high-confidence way to reproduce the issue?

Not applicable; this is a QA coverage wrapper rather than a bug report. Source review confirms the existing script and docs define the behavior being linked into QA Lab.

Is this the best way to solve the issue?

Yes; a script-backed scenario YAML is the supported QA Lab path for existing script producers, and it avoids duplicating Tool Search E2E logic before a first-class temporary-plugin fixture exists.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 119dc4bd821e.

Label changes

Label justifications:

  • P3: This is low-risk QA coverage metadata/linking work with no runtime product behavior change.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: This maintainer-labeled PR is not subject to the external-contributor real behavior proof gate; the PR body still includes targeted terminal and QA-suite evidence.
Evidence reviewed

PR surface:

Other +28. Total +28 across 1 file.

View PR surface stats
Area Files Added Removed Net
Source 0 0 0 0
Tests 0 0 0 0
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 1 28 0 +28
Total 1 28 0 +28

What I checked:

Likely related people:

  • RomneyDa: Authored the current PR and recent merged QA scenario coverage work that uses the same wrapper pattern. (role: recent QA coverage contributor; confidence: high; commits: 1061882269db, ff6289049084, 119dc4bd821e; files: qa/scenarios/runtime/tool-search-gateway-e2e.yaml, qa/scenarios/runtime/openai-compatible-chat-tools.yaml, qa/scenarios/runtime/openwebui-openai-compatible.yaml)
  • steipete: Authored the merged Tool Search PR that introduced the core runtime, docs, and the gateway E2E script this wrapper links. (role: Tool Search feature owner by history; confidence: high; commits: b85cb4e8d13e, b78604a0774f, 15b26be8daf0; files: src/agents/tool-search.ts, scripts/tool-search-gateway-e2e.ts, docs/tools/tool-search.md)
  • pfrederiksen: Opened adjacent maintenance work for the Tool Search gateway E2E script and paired timeout issue, which may be useful context even though that PR was not merged. (role: adjacent script contributor; confidence: medium; commits: 58011ff347fb; files: scripts/tool-search-gateway-e2e.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. labels Jun 28, 2026
@RomneyDa
RomneyDa merged commit 39c4c02 into main Jun 28, 2026
165 of 174 checks passed
@RomneyDa
RomneyDa deleted the dallin/qa-tool-search-gateway-coverage branch June 28, 2026 11:28
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 29, 2026
QiuYuang pushed a commit to QiuYuang/openclaw that referenced this pull request Jul 1, 2026
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 1, 2026
chenyangjun-xy pushed a commit to chenyangjun-xy/openclaw that referenced this pull request Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extensions: qa-lab maintainer Maintainer-authored PR P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: XS status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant