Skip to content

fix(agents): keep exec visible for lean local models#87587

Closed
vincentkoc wants to merge 1 commit into
mainfrom
small-model-lean-routing
Closed

fix(agents): keep exec visible for lean local models#87587
vincentkoc wants to merge 1 commit into
mainfrom
small-model-lean-routing

Conversation

@vincentkoc

@vincentkoc vincentkoc commented May 28, 2026

Copy link
Copy Markdown
Member

Refs #86632.

Summary

Lean local-model mode now keeps exec directly visible when Tool Search compacts the rest of the tool catalog. This gives small coding-tuned local models an obvious shell fallback for simple live-data or workspace probes instead of forcing them to search for a domain-specific tool and stop when none exists.

This deliberately does not add a cmd alias or automatically route weather/live-data requests through shell; those are broader behavior and security decisions.

Verification

  • node scripts/run-vitest.mjs src/agents/local-model-lean.test.ts src/agents/tool-search.test.ts
  • node scripts/run-vitest.mjs src/agents/embedded-agent-runner/run/attempt.tool-search-run-plan.test.ts
  • post-rebase: node scripts/run-vitest.mjs src/agents/local-model-lean.test.ts src/agents/tool-search.test.ts src/agents/embedded-agent-runner/run/attempt.tool-search-run-plan.test.ts
  • node scripts/format-docs.mjs docs/concepts/experimental-features.md docs/gateway/local-models.md docs/providers/ollama.md
  • git diff --check
  • AWS Crabbox: corepack pnpm check:changed on cbx_58160a36c5b3 / run_e0befbd1db32 (exit 0)
  • autoreview: clean, no accepted/actionable findings

Real behavior proof

Behavior addressed: lean local-model Tool Search no longer hides exec, so small coding models can directly discover shell execution while heavier/default tools remain removed or cataloged.
Real environment tested: local linked OpenClaw worktree plus AWS Crabbox Linux runner cbx_58160a36c5b3.
Exact steps or command run after this patch: node scripts/run-vitest.mjs src/agents/local-model-lean.test.ts src/agents/tool-search.test.ts src/agents/embedded-agent-runner/run/attempt.tool-search-run-plan.test.ts; corepack pnpm check:changed remotely through Crabbox.
Evidence after fix: focused tests passed locally after rebase; Crabbox changed gate completed successfully with exit 0.
Observed result after fix: Tool Search keeps predicate-excluded tools visible, and lean mode passes that predicate so exec remains in the direct tool list.
What was not tested: a live Ollama/Qwen weather prompt; this patch is the narrow catalog/visibility fix, not an automatic live-data routing policy.

@vincentkoc vincentkoc self-assigned this May 28, 2026
@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation gateway Gateway runtime agents Agent runtime and tooling size: S maintainer Maintainer-authored PR labels May 28, 2026
@clawsweeper

clawsweeper Bot commented May 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: found issues before merge. Reviewed June 21, 2026, 4:18 AM ET / 08:18 UTC.

Summary
The PR keeps exec out of the Tool Search catalog for local-model lean runs, wires the predicate through the embedded runner, and updates focused tests and local-model docs.

PR surface: Source +14, Tests +35, Docs +3. Total +52 across 8 files.

Reproducibility: no. live Ollama/Qwen weather-session reproduction was run here. Source inspection does reproduce the narrow catalog mechanism: current main can catalog exec behind Tool Search in lean mode unless a direct visibility exception is added.

Review metrics: 1 noteworthy metric.

  • Direct-tool visibility exceptions: 1 added (exec). This is a model-facing tool-surface change for opt-in lean local-model runs with Tool Search enabled.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #86632
Summary: The linked issue is the canonical user report; this PR is a candidate fix for the narrow direct-exec visibility portion and intentionally leaves cmd aliasing and automatic live-data routing separate.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🌊 off-meta tidepool
Patch quality: 🦐 gold shrimp
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Rebase onto current main and preserve the merged lean Tool Search defaults and directory-mode behavior.
  • [P2] Either implement the direct-exec exception for directory mode or narrow the docs to tools mode.

Risk before merge

  • [P1] Merging would make shell execution more prominent for users who already opted into lean local-model mode with Tool Search, even though exec permissions and approvals are unchanged.
  • [P1] The branch predates current-main lean Tool Search defaults and directory-mode docs, so a rebase must preserve those behaviors instead of restoring the older runtime/docs shape.
  • [P1] The PR proof covers catalog visibility and changed gates, not a live Ollama/Qwen weather prompt, a cmd alias, or automatic live-data-to-shell routing.

Maintainer options:

  1. Rebase and align Tool Search modes (recommended)
    Update the branch on current main so the direct-exec rule either covers directory mode too or the public docs explicitly limit the behavior to tools mode.
  2. Accept tools-mode-only behavior
    Maintainers may intentionally keep the implementation scoped to lean's default tools mode, but the docs should not promise broader Tool Search behavior.
  3. Hold the direct-exec direction
    Pause or close this PR if maintainers do not want lean mode to make shell execution more prominent for opt-in local-model users.

Next step before merge

  • [P2] Manual maintainer review is needed because this is a draft, maintainer-labeled PR whose remaining blockers are current-main alignment and shell-tool visibility acceptance, not a narrow safe automation repair.

Security
Cleared: No dependency, workflow, secret, permission, or approval-gate change was found; the shell visibility question remains a maintainer product/security tradeoff rather than a new bypass in the diff.

Review findings

  • [P2] Narrow the Tool Search promise or cover directory mode — docs/concepts/experimental-features.md:33
Review details

Best possible solution:

Rebase onto current main, align direct-exec visibility and docs across supported Tool Search modes, and land only after maintainers accept the opt-in shell visibility tradeoff while keeping cmd aliasing and automatic live-data routing separate.

Do we have a high-confidence way to reproduce the issue?

No live Ollama/Qwen weather-session reproduction was run here. Source inspection does reproduce the narrow catalog mechanism: current main can catalog exec behind Tool Search in lean mode unless a direct visibility exception is added.

Is this the best way to solve the issue?

No, not as currently aligned to current main. The predicate is a good narrow seam for tools mode, but the docs overclaim all Tool Search modes and the shell visibility tradeoff still needs maintainer acceptance.

Full review comments:

  • [P2] Narrow the Tool Search promise or cover directory mode — docs/concepts/experimental-features.md:33
    The new docs promise that lean mode keeps exec direct when Tool Search is enabled, but current main has a separate directory mode path through applyToolSchemaDirectoryCatalog. After rebase, either pass the same direct-exec rule through directory mode or scope the docs to the tools-mode catalog path.
    Confidence: 0.86

Overall correctness: patch is incorrect
Overall confidence: 0.86

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 9940110b881b.

Label changes

Label justifications:

  • P2: The PR addresses a bounded local-model agent-routing problem with limited blast radius to opt-in lean local-model workflows.
  • merge-risk: 🚨 compatibility: The patch changes the visible tool surface for existing lean local-model plus Tool Search users by making exec easier for the model to choose.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🌊 off-meta tidepool and patch quality is 🦐 gold shrimp.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Not applicable: The external-contributor proof gate does not apply to this maintainer-associated draft PR; the body includes focused tests and Crabbox changed-gate proof for the narrow catalog path, but not a live Ollama/Qwen prompt.
Evidence reviewed

PR surface:

Source +14, Tests +35, Docs +3. Total +52 across 8 files.

View PR surface stats
Area Files Added Removed Net
Source 3 19 5 +14
Tests 2 36 1 +35
Docs 3 8 5 +3
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 8 63 11 +52

What I checked:

Likely related people:

  • vincentkoc: Authored this PR and the merged lean Tool Search default work that current main now contains. (role: recent area contributor; confidence: high; commits: e43eff053a5b, 6153fb5ecb47, 8b03fd1f5f83; files: src/agents/local-model-lean.ts, src/agents/embedded-agent-runner/run/attempt.ts, docs/concepts/experimental-features.md)
  • steipete: Introduced the core Tool Search runtime and runner wiring that this PR extends. (role: feature introducer; confidence: high; commits: b85cb4e8d13e, a72007df0144, 15b26be8daf0; files: src/agents/tool-search.ts, src/agents/embedded-agent-runner/run/attempt.ts, docs/tools/tool-search.md)
  • dutifulbob: Introduced per-agent local-model lean mode and its initial helper/test/docs surface. (role: feature introducer; confidence: high; commits: 13c97c5a8d04, bd7dba62a74c; files: src/agents/local-model-lean.ts, src/agents/local-model-lean.test.ts, docs/concepts/experimental-features.md)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels May 28, 2026
@clawsweeper

clawsweeper Bot commented May 28, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper PR egg: ✨ hatched 🥚 common Frosted Proofling. Rarity: 🥚 common. Trait: sparkles near resolved comments.

Details

Share on X: post this hatch
Copy: My PR egg hatched a 🥚 common Frosted Proofling in ClawSweeper.
Hatchability:

  • Merged PRs are hatchable.
  • Open PRs are hatchable when they are status: 👀 ready for maintainer look, status: 🚀 automerge armed, or labeled clawsweeper:automerge.
  • Closed unmerged PRs are hatchable only when one of those hatchable labels is still present in the durable record.

About:

  • Eggs appear after real-behavior proof passes. They are collectible flavor only.
  • Review momentum changes the shell state: follow-up work warms it, re-review makes it wobble, and a clean final review lets it hatch.
  • The hatch is seeded from this repository and PR number, so the same PR keeps the same creature; the reviewed head SHA can only change safe visual details.
  • Rarity is just collectible sparkle: 🥚 common, 🌱 uncommon, 💎 rare, ✨ glimmer, and 🌈 legendary.

@clawsweeper clawsweeper Bot added the P2 Normal backlog priority with limited blast radius. label May 28, 2026
@clawsweeper clawsweeper Bot added the merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. label May 29, 2026
@clawsweeper clawsweeper Bot added rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels May 29, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. labels Jun 15, 2026
@clawsweeper clawsweeper Bot mentioned this pull request Jun 19, 2026
Open
@clawsweeper clawsweeper Bot added status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 20, 2026
@clawsweeper clawsweeper Bot mentioned this pull request Jun 22, 2026
Draft
@steipete steipete mentioned this pull request Jul 7, 2026
Merged
@steipete

steipete commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Thanks @vincentkoc — the maintained current-main replay landed in #101607 as 7c0a7c8be210c7d95b0b4e6b4cf0126e5c2e7e44.

The replay preserves your co-authorship and diagnosis, narrows the exception to policy-filtered exec in structured Tool Search tools mode, covers both runtime tool-surface owners, and adds explicit code-mode regression proof. The original branch was stale and had maintainer edits disabled, so it could not be safely refreshed in place.

Closing this PR as superseded by the landed replay. For future PRs, enabling GitHub's Allow edits by maintainers lets us make these fixups directly on the contributor branch. Thank you for identifying the lean local-model tool-visibility gap.

@steipete steipete closed this Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@vincentkoc vincentkoc

Labels

agents Agent runtime and tooling docs Improvements or additions to documentation gateway Gateway runtime maintainer Maintainer-authored PR merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. size: S status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@vincentkoc @steipete @RomneyDa