Skip to content

improve: reduce Code Mode catalog discovery turns#109651

Merged
steipete merged 1 commit into
mainfrom
codex/tool-surface-performance-followup
Jul 17, 2026
Merged

improve: reduce Code Mode catalog discovery turns#109651
steipete merged 1 commit into
mainfrom
codex/tool-surface-performance-followup

Conversation

@steipete

Copy link
Copy Markdown
Contributor

Related: #109288
Follow-up to #109596

AI-assisted: Yes (Codex)

What Problem This Solves

Code Mode kept its compact tool catalog entirely inside the guest runtime. Models therefore spent an initial exec turn discovering exact tool ids, or guessed nonexistent ids and used more turns to recover, before useful work began.

Why This Change Was Made

The model-visible exec description now carries a deterministic, 8,000-character quick index of exact OpenClaw/plugin catalog ids and compact input hints. Full schemas, descriptions, MCP entries, and overflow entries remain deferred to guest-side catalog lookup, which preserves the bounded prompt and avoids exposing adversarial catalog prose.

The index marks result schemas as unknown. When output fields are not documented, guidance asks the model to return the raw value first and process the observed shape in a later exec instead of guessing fields. A shared compact-entry formatter keeps the prompt index and guest catalog representation aligned.

User Impact

Code Mode can begin common tool workflows without a separate catalog-discovery round trip. Across the 72-tool adversarial live harness, it used fewer turns, tool calls, tokens, and wall-clock time while improving validated task success and keeping decoy calls at zero.

Evidence

Live multi-provider Code Mode matrix, four validated tasks per provider, 72-tool catalog with adversarial decoys. Baseline is merged #109596; after is this branch. These are single live matrix runs, so provider latency and exact success counts remain stochastic.

Metric Before After Change
Validated tasks 9/12 11/12 +2
Model turns 49 29 -41%
Tool calls 37 17 -54%
Decoy calls 0 0 unchanged
Total latency 133.2s 81.4s -39%
Input tokens 46,149 38,827 -16%
Output tokens 4,877 2,388 -51%

Provider aggregates:

Provider Success before/after Turns before/after Latency before/after
OpenAI 3/4 -> 4/4 12 -> 10 28.8s -> 27.8s
Anthropic 4/4 -> 4/4 16 -> 11 45.7s -> 35.5s
Google 2/4 -> 3/4 21 -> 8 58.7s -> 18.1s

The nonexistent-tool recovery task passed on all three providers. No provider called an adversarial decoy.

Validation:

  • node scripts/run-vitest.mjs src/agents/code-mode.test.ts src/agents/tool-search.test.ts: 129 tests passed.
  • node scripts/check-changed.mjs -- src/agents/code-mode.ts src/agents/code-mode.test.ts src/agents/tool-search.ts docs/reference/code-mode.md: delegated Testbox run passed, including core/test typechecks, formatting, lint, API surface, dependency, boundary, and runtime guards.
  • Autoreview: clean, no accepted/actionable findings, confidence 0.93.
  • Source-blind behavior contract: satisfied; success increased, turns decreased, recovery passed, and decoy calls remained zero.

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation agents Agent runtime and tooling size: S maintainer Maintainer-authored PR labels Jul 17, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. labels Jul 17, 2026
@clawsweeper

clawsweeper Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 17, 2026, 12:52 AM ET / 04:52 UTC.

Summary
The PR adds a bounded Code Mode quick index of exact OpenClaw/plugin tool IDs and compact input hints, reuses the catalog formatter, and updates tests and documentation.

PR surface: Source +58, Tests +82, Docs +15. Total +155 across 4 files.

Reproducibility: not applicable. as a conventional bug reproduction; the performance opportunity is demonstrated through an exact-branch live multi-provider benchmark against the merged predecessor.

Review metrics: 1 noteworthy metric.

  • Prompt index budget: 1 model-visible index capped at 8,000 characters. The explicit cap bounds the central prompt-size and provider-token tradeoff introduced by the optimization.

Stored data model
Persistent data-model change detected: serialized state: src/agents/code-mode.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

  • [P1] The live provider matrix is a single run, so the magnitude of the performance gains is directional even though all three providers exercised the changed path successfully.
  • [P1] The PR head is behind current main; exact-head checks and a refreshed three-way diff should complete before merge, although no concrete stale-base defect is currently shown.

Maintainer options:

  1. Decide the mitigation before merge
    Land the shared, bounded native-tool index after refreshing it onto current main and confirming exact-head checks, while retaining deferred descriptions, full schemas, MCP entries, and overflow lookup inside the guest runtime.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • No ClawSweeper repair is indicated; the PR has no concrete patch defect and should proceed through normal exact-head checks and maintainer merge review.

Security
Cleared: The diff adds no dependency, workflow, permission, secret, download, or package-resolution changes and keeps descriptions and MCP metadata outside the model-visible index.

Review details

Best possible solution:

Land the shared, bounded native-tool index after refreshing it onto current main and confirming exact-head checks, while retaining deferred descriptions, full schemas, MCP entries, and overflow lookup inside the guest runtime.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a conventional bug reproduction; the performance opportunity is demonstrated through an exact-branch live multi-provider benchmark against the merged predecessor.

Is this the best way to solve the issue?

Yes. Reusing the compact catalog formatter and exposing only a deterministic bounded index is narrower and safer than adding configuration or moving full schemas and untrusted descriptions into the model prompt.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against a77c0770ffe6.

Label changes

Label changes:

  • add P2: This is a well-supported core agent performance improvement with limited blast radius and measurable workflow benefit.
  • add proof: sufficient: Contributor real behavior proof is sufficient. The PR body contains after-fix live output from a real adversarial multi-provider harness showing improved success and substantial reductions in turns, calls, tokens, and latency with zero decoy calls.
  • add rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body contains after-fix live output from a real adversarial multi-provider harness showing improved success and substantial reductions in turns, calls, tokens, and latency with zero decoy calls.

Label justifications:

  • P2: This is a well-supported core agent performance improvement with limited blast radius and measurable workflow benefit.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body contains after-fix live output from a real adversarial multi-provider harness showing improved success and substantial reductions in turns, calls, tokens, and latency with zero decoy calls.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body contains after-fix live output from a real adversarial multi-provider harness showing improved success and substantial reductions in turns, calls, tokens, and latency with zero decoy calls.
Evidence reviewed

PR surface:

Source +58, Tests +82, Docs +15. Total +155 across 4 files.

View PR surface stats
Area Files Added Removed Net
Source 2 68 10 +58
Tests 1 83 1 +82
Docs 1 20 5 +15
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 4 171 16 +155

What I checked:

  • Current-main gap: At current main, Code Mode keeps compact catalog metadata inside the guest runtime; the PR inserts the new prompt-index formatter after this point, so its central optimization is not already implemented. (src/agents/code-mode.ts:1269, a77c0770ffe6)
  • Bounded prompt implementation: The branch constructs a deterministic OpenClaw/plugin-only index, marks result schemas unknown, omits descriptions, and limits the added index to 8,000 characters with an overflow fallback. (src/agents/code-mode.ts:1271, 271457894acc)
  • Shared representation: The branch exports and reuses the existing compact catalog-entry formatter rather than creating a second input-hint implementation, keeping guest lookup and prompt metadata aligned. (src/agents/tool-search.ts:1261, 271457894acc)
  • Regression coverage: Added tests cover exact IDs, deterministic ordering, deferred descriptions, unknown-output guidance, MCP exclusion, and bounded overflow behavior. (src/agents/code-mode.test.ts:422, 271457894acc)
  • Documented contract: The documentation distinguishes hidden standalone tools from the bounded native-tool index and states that full schemas, descriptions, MCP entries, and overflow entries remain guest-side. Public docs: docs/reference/code-mode.md. (docs/reference/code-mode.md:39, 271457894acc)
  • Live behavior proof: The PR reports an exact-branch, 72-tool adversarial matrix across OpenAI, Anthropic, and Google: validated tasks rose from 9/12 to 11/12, model turns fell from 49 to 29, and no decoy tool was called. (271457894acc)

Likely related people:

  • steipete: Authored the merged compact-catalog optimization and this follow-up, giving the strongest recent history on the exact Code Mode performance path. (role: recent feature owner; confidence: high; commits: 57cc2ca308c1, 271457894acc; files: src/agents/code-mode.ts, src/agents/tool-search.ts, src/agents/code-mode.test.ts)
  • Kaspre: Recent merged Code Mode work credited this contributor for model-visible exec-schema constraints and adjacent runtime error and hook behavior, making them a useful secondary reviewer for prompt-contract changes. (role: earlier Code Mode contributor; confidence: medium; files: src/agents/code-mode.ts, src/agents/code-mode.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@steipete steipete self-assigned this Jul 17, 2026
@steipete
steipete merged commit c742cae into main Jul 17, 2026
178 of 188 checks passed
@steipete
steipete deleted the codex/tool-surface-performance-followup branch July 17, 2026 06:09
@steipete

Copy link
Copy Markdown
Contributor Author

Merged via squash.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling docs Improvements or additions to documentation maintainer Maintainer-authored PR P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant