Skip to content

[Feature]: on-demand MCP tool loading via tool_search (deferred tool activation) #3095

Closed

Description

@richin13

Summary

Load MCP tool schemas on demand instead of eagerly, reducing system prompt size and context waste across all providers

Problem statement

Every configured MCP server currently loads ALL its tools eagerly at startup. Full tool schemas (name, description, JSON parameter schema) are serialized into every LLM API call and system prompt. With multiple MCP servers, this adds dozens or hundreds of tool definitions, wasting context window tokens on tools that may never be used in a given conversation. For example, a setup with 3 MCP servers exposing 50 tools total could add ~77K tokens of tool schema to every turn, even when only 2-3 tools are actually needed. This degrades both cost efficiency and model accuracy (more noise in context = worse tool selection).

Proposed solution

Implement Claude Code-style deferred tool loading with two tiers:

  • Deferred: MCP tools are listed by name and short description only in the system prompt inside an <available-deferred-tools> section. They are NOT sent as native tool definitions.
  • Activated: When the LLM needs an MCP tool, it first calls a built-in tool_search tool (always present when deferred MCP tools exist). This fetches the full schema, registers the tool in a per-conversation ActivatedToolSet, and returns the full definition so the LLM can call it on the next turn.

Controlled by mcp.deferred_loading config key (default true). Setting false preserves current eager behavior exactly.

Key components:

  • DeferredMcpToolStub / DeferredMcpToolSet: lightweight name+description stubs for system prompt listing and search
  • ActivatedToolSet: per-conversation mutable state tracking tools activated during a conversation
  • ToolSearchTool: built-in tool implementing Tool trait, supports select:name1,name2 for exact match and keyword search
  • Per-iteration tool_specs recomputation in the agent loop to pick up newly activated tools
  • Helpful error message when LLM tries to call a deferred tool without activating it first

Reference: Anthropic Advanced Tool Use describes this pattern and reports context reduction from ~77K to ~8.7K tokens with accuracy improvements.

Non-goals / out of scope

  • Deferred loading for native (non-MCP) tools. These are always eagerly loaded as today.
  • Persistent activation across conversations. Activated tools are scoped to a single conversation turn.
  • Automatic activation (LLM must explicitly call tool_search).
  • Changes to MCP server connection lifecycle. Servers are still connected at startup; only schema delivery to the LLM is deferred.

Alternatives considered

  • Signature-based approach: Adding deferred_tools and activated_tools parameters directly to run_tool_call_loop. Rejected because the function already has 15 parameters and this would require updating ~20+ test call sites. Task-local variables (consistent with existing TOOL_LOOP_* pattern) achieve the same result with zero signature churn.
  • Combined registry rebuild: Building a new Vec<Box<dyn Tool>> each iteration by cloning the entire static registry plus activated tools. Rejected because Box<dyn Tool> is not cloneable. Instead, execution functions accept an extra_tools slice that is searched alongside the static registry.
  • Automatic activation on first call: Transparently activating deferred tools when the LLM calls them without tool_search. Rejected because it hides the activation step from the LLM, preventing it from seeing the full schema before crafting arguments.

Acceptance criteria

  1. mcp.deferred_loading = true (default): MCP tools appear only as names in the system prompt, tool_search is registered as a built-in tool, calling tool_search activates tools and makes them callable on subsequent turns.
  2. mcp.deferred_loading = false: Current eager behavior preserved exactly. No tool_search tool, no <available-deferred-tools> section.
  3. Calling a deferred tool without activation returns a helpful error directing the LLM to use tool_search.
  4. All existing tests pass with no regressions. Non-MCP tool behavior completely unchanged.
  5. docs/config-reference.md documents mcp.deferred_loading.

Architecture impact

  • src/tools/ — new mcp_deferred and tool_search modules, McpRegistry::empty() constructor, module registration
  • src/config/schema.rs — McpConfig.deferred_loading field
  • src/agent/loop_.rs — task-local DeferredToolContext, per-iteration tool_specs recomputation, deferred-not-activated error path
  • src/agent/loop_/execution.rs — extra_tools parameter on find_tool / execute_tools_* functions
  • src/agent/prompt.rs — build_deferred_tools_section() rendering
  • src/channels/mod.rs — conditional deferred vs eager MCP wiring, system prompt injection, ChannelRuntimeContext field, task-local scoping

Risk and rollback

No response

Breaking change?

No

Data hygiene checks

  • I removed personal/sensitive data from examples, payloads, and logs.
  • I used neutral, project-focused wording and placeholders.

Activity

  1. mykolademyanov commented on Mar 10, 2026

    @mykolademyanov

    Once agents start interacting with tools or external systems, debugging usually becomes harder than expected.

    One pattern that helps is separating reasoning from execution.
    The model proposes actions, while another layer actually executes them, handles retries and records state.

    Otherwise failures often get hidden inside the agent loop.

    Short overview of this runtime pattern:
    https://agentpatterns.tech/en/architecture/agent-runtime

  2. Dave-London commented on Mar 10, 2026

    @Dave-London

    Deferred loading solves the schema side of the token problem nicely. The output side matters too — a single verbose tool call can burn thousands of tokens of formatting. We address this by returning structured JSON instead of raw CLI text, typically 70-95% fewer output tokens. https://github.com/Dave-London/Pare

  3. added a commit that references this issue on Mar 13, 2026
    70a2ddf
  4. added a commit that references this issue on Mar 13, 2026
    ef770f1
  5. added a commit that references this issue on Mar 18, 2026
    e853dd5
  6. added a commit that references this issue on Mar 19, 2026
    673fdfd
  7. added a commit that references this issue on Oct 9, 2026
    b0d1a08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions