test: link Tool Search gateway QA coverage#97374
Conversation
|
Codex review: needs maintainer review before merge. Reviewed June 28, 2026, 1:41 AM ET / 05:41 UTC. Summary PR surface: Other +28. Total +28 across 1 file. Reproducibility: not applicable. this is a QA coverage wrapper rather than a bug report. Source review confirms the existing script and docs define the behavior being linked into QA Lab. Review metrics: none identified. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Next step before merge
Security Review detailsBest possible solution: Merge the focused scenario wrapper after normal maintainer review and required checks, leaving the broader temporary-plugin fixture migration as a separate QA Lab follow-up. Do we have a high-confidence way to reproduce the issue? Not applicable; this is a QA coverage wrapper rather than a bug report. Source review confirms the existing script and docs define the behavior being linked into QA Lab. Is this the best way to solve the issue? Yes; a script-backed scenario YAML is the supported QA Lab path for existing script producers, and it avoids duplicating Tool Search E2E logic before a first-class temporary-plugin fixture exists. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 119dc4bd821e. Label changesLabel justifications:
Evidence reviewedPR surface: Other +28. Total +28 across 1 file. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
What Problem This Solves
Batch 2 of the QA maturity linking work had an existing Tool Search gateway E2E script, but no QA Lab scenario wrapper. That meant the evidence runner could not count this existing proof in the scenario inventory.
This PR does not change taxonomy and does not add new product behavior.
Why This Change Was Made
This adds one native QA scenario wrapper for
scripts/tool-search-gateway-e2e.ts.Primary coverage claimed:
tool-search-gateway-e2escripts/tool-search-gateway-e2e.tsruntime.hosted-tool-useWhy primary: the script starts the QA mock OpenAI provider and a real Gateway, creates a temporary fake plugin with a large tool catalog, calls
/v1/responses, and asserts both direct mode and Tool Search code mode invoke the selected plugin tool. It also verifies compact mode exposes only thetool_search_codebridge to the provider while still producing the plugin tool result.Secondary coverage kept:
plugins.plugin-toolstools.invocationIntentionally not claimed:
gateway.tool-and-skill-apis. I checked it, but this script does not call thecommands.list,tools.*, orskills.*Gateway RPC surface described by that taxonomy ID.User Impact
The QA scenario inventory can now run and score the existing Tool Search gateway E2E proof through QA Lab. Operators get evidence for existing hosted tool-use behavior without changing runtime code.
Follow-up note: fully deleting
scripts/tool-search-gateway-e2e.tsis plausible, but not in this PR. Today the QA flow DSL can run gateway calls and assertions, but it does not have a clean first-class primitive for staging a temporary plugin directory before gateway startup. The smallest durable migration would be a reusable QA Lab fixture/helper for temporary plugin tool catalogs, then a flow scenario can replace the standalone script.Evidence
node scripts/run-vitest.mjs test/scripts/tool-search-gateway-e2e.test.ts extensions/qa-lab/src/scenario-catalog.test.ts extensions/qa-lab/src/coverage-report.test.ts extensions/qa-lab/src/scorecard-evidence.test.tspnpm openclaw qa suite --provider-mode mock-openai --scenario tool-search-gateway-e2e --output-dir .artifacts/qa-e2e/tool-search-gateway-coverage.artifacts/qa-e2e/tool-search-gateway-coverage/qa-evidence.jsongit diff --check -- qa/scenarios/runtime/tool-search-gateway-e2e.yaml.agents/skills/autoreview/scripts/autoreview --mode localgateway.tool-and-skill-apis; accepted and removed