As AI agents gain access to more tools and capabilities, a critical challenge emerges: the more tools an LLM can choose from, the less accurate it becomes at selecting the right one. This inverse relationship between tool quantity and selection accuracy fundamentally limits how many capabilities we can expose to a single agent.
This guide examines proven strategies to overcome this limitation, drawing from publicly available research and real-world implementations from GitHub, Anthropic, and Cloudflare. We explore both techniques that help LLMs better understand available tools and approaches that strategically reduce the number of tools presented at any given time.
- The Challenge
- Solution Approaches Overview
- Semantic Solutions (Help LLMs distinguish between tools)
- Tools Reduction Solutions (Reduce the number of tools)
- API Calling vs Tool Calling Approach
- Summary
- References
Three fundamental constraints shape how we expose tools to LLMs:
| Constraint | Description | Impact |
|---|---|---|
| Platform Limitations | Most LLMs support max 128 tools through structured API calling | Hard ceiling on capabilities |
| Accuracy Degradation | Tool selection accuracy drops progressively beyond ~10 tools | Practical ceiling far below platform limits |
| Resource Overhead | Each tool consumes tokens in system prompts | Higher costs, reduced context for tasks |
Key Insight: The tension isn't just about how many tools to expose, but achieving the right balance between capability breadth and selection accuracy.
Two main categories of solutions address the tool selection accuracy challenge:
Help the LLM better understand and distinguish between tools based on their semantic meaning.
- Namespaces: Organize tools using hierarchical naming
- Additional Strategies: Enhanced documentation, few-shot learning, tool ranking, and more
- Hierarchical Agents: Distribute tools across specialized agents
Limit the actual number of tools the LLM needs to choose from at any given moment.
- Abstraction: Combine multiple operations into higher-level business tools
- Grouping and Filtering: Dynamically present only relevant tool subsets
Concept: Organize tools using hierarchical naming conventions with dot notation or path-like structures.
order.management.create_orderorder.management.delete_orderorder.fulfillment.ship_orderorder.fulfillment.track_shipmentorder.payment.process_paymentorder.payment.refund_payment
Shown in hierarchical view as below:
order/
├── management/
│ ├── create_order
│ └── delete_order
├── fulfillment/
│ ├── ship_order
│ └── track_shipment
└── payment/
├── process_payment
└── refund_payment
LLMs are designed to predict the next token. Namespacing aligns perfectly with this capability. When processing order.payment.process_payment:
- First token:
order→ establishes the domain - Second token:
payment→ narrows to payment operations - Third token:
process_payment→ identifies the specific action
This hierarchical structure provides clear semantic grouping and helps the LLM understand relationships between tools.
-
Attention Mechanism Alignment: LLMs use attention mechanisms to identify patterns and relationships. Namespaced tools create clear structural patterns (domain.subdomain.action) that the attention layers can easily recognize and differentiate, reducing confusion between similarly-named tools.
-
Semantic Clustering in Embedding Space: During training, LLMs learn that words appearing in similar contexts have related meanings. Namespaces explicitly group related tools together (e.g., all
order.payment.*tools), which helps the model's internal representations cluster semantically related operations, making the right tool more likely to have higher activation. -
Reduced Ambiguity: Without namespaces, tools like
create_order,create_customer,create_productall start with "create" and look similar to the model. With namespaces likeorder.management.create_order,customer.management.create_customer, the domain context is embedded in the tool name itself, drastically reducing ambiguity. -
Prefix-Based Pruning: When the LLM begins generating a tool name, the namespace prefix acts as a constraint. Once it generates
order.payment., it has effectively pruned all tools outside that namespace from consideration, narrowing the search space dramatically and improving selection accuracy. -
Training Data Similarity: Most codebases and APIs in the training data use similar hierarchical conventions (package.module.function, namespace.class.method), so LLMs have seen countless examples of this pattern and learned to process it effectively.
-
Compositional Understanding: LLMs excel at compositional semantics - understanding meaning built from parts. A namespaced tool name like
inventory.warehouse.check_stockis compositional: the model understands "inventory" + "warehouse" + "check stock" as a natural combination, whereas isolated tool names require the model to infer context from descriptions alone.
| Benefit | Description |
|---|---|
| Improved Context | Namespace prefix provides immediate domain context |
| Better Differentiation | Similar tools in different domains easily distinguished |
| Scalability | Organizational structure remains clear as tools grow |
| Single Agent | Works within one LLM call without latency/cost overhead |
| Natural Token Prediction | Aligns with LLM's core sequential processing strength |
- Requires consistent naming conventions across all tools
- Tool names become longer, slightly increasing token usage
- Works best when combined with good descriptions that reinforce the namespace hierarchy
Beyond namespacing, several complementary techniques enhance tool selection accuracy through better understanding:
Provide rich contextual information with structured metadata:
{
"name": "process_order_shipment",
"when_to_use": "When an order is ready to be shipped to the customer",
"when_not_to_use": "Don't use for order creation or already shipped orders",
"categories": ["order_management", "fulfillment", "logistics"],
"prerequisites": ["order in 'confirmed' status", "payment completed"],
"complexity": "medium"
}Include 2-3 examples of successful tool usage in the system prompt:
Example: User wants to "ship an order and notify customer"
→ Use: get_order_details(order_id) → process_shipment(order_id) → send_notification(customer_id)
Why tool order matters: LLMs favor tools presented first (primacy effect) and last (recency effect). Tools in early positions are selected 2-3× more often than those beyond position 10.
Impact: Tool selection accuracy varies by 10-15% based on ordering alone. Frequency-based ordering improves success rates by 5-12%.
Best practices:
- Place core/frequently-used tools first
- Dynamically reorder based on query context
- Prioritize tools with higher success rates
- Boost recently-used tools within current session
Example:
Query: "Complete order shipment"
Reordered: [ship_order, track_shipment, send_notification, ...]
Not: [create_order, cancel_order, ship_order, ...]
- Track individual user's tool preferences and usage patterns
- Adapt based on user's role (developer, designer, manager)
- Allow users to mark favorite tools or hide irrelevant ones
- Monitor tool selection accuracy in production
- Track failure rates and error patterns
- A/B test different descriptions, groupings, or orderings
- Fine-tune models on domain-specific tool usage patterns
Define explicit relationships between tools:
Tool: create_order
Related tools:
- Next steps: process_payment, update_order_status
- Requires: validate_customer
- Conflicts with: cancel_order (on same order)
- Intent Classification: Pre-classify queries into categories, activate only relevant tool categories
- Entity Extraction: Extract key entities, pre-filter tools compatible with detected entities
Concept: Create a hierarchy of specialized agents where each handles a specific domain, similar to Domain-Driven Design (DDD) in microservices.
Note: This approach combines semantic routing with tools reduction through abstraction—each agent only sees relevant tools for their domain.
Instead of a single monolithic agent, the system distributes responsibility across multiple specialized agents. Requests are progressively routed from general orchestration agents down to specialized domain agents that possess the specific tools needed.
Orchestration Agent
└── Ordering Agent
├── Order Management Agent
│ ├── create_order
│ └── delete_order
├── Order Fulfillment Agent
│ ├── ship_order
│ └── track_shipment
└── Order Payment Agent
├── process_payment
└── refund_payment
The goal is to prevent any single agent from being overloaded with tools. Requests flow from top to bottom:
- Orchestration Agent: Routes user request to appropriate domain agent
- Domain Agent: Routes to appropriate sub-domain agent
- Specialist Agent: Executes the actual tool call
-
Cognitive Load Reduction: By distributing tools across multiple specialized agents, each individual LLM operates with a smaller, more focused tool set. This mirrors human cognitive limits - specialists perform better in narrow domains than generalists trying to handle everything.
-
Context Specialization: Each agent can maintain domain-specific context, system prompts, and examples optimized for its specialized area. An "Order Payment Agent" can have payment-specific instructions, validation rules, and error handling that would clutter a general-purpose agent.
-
Parallel Processing Potential: Multiple leaf agents can theoretically process independent requests in parallel, improving throughput in high-load scenarios (though this adds complexity).
-
Tool Set Isolation: Changes to tools in one domain don't affect other domains. Adding new payment tools doesn't increase the cognitive load on the fulfillment or management agents.
-
Explicit Reasoning Chain: The routing decisions create an audit trail showing how the system arrived at a particular tool selection, which aids debugging and system improvement.
-
Domain Expertise Modeling: Different agents can use different base models, prompts, or even fine-tuned versions optimized for their specific domain, potentially improving accuracy within each specialization.
Critical Consideration: Multiple LLMs in a chain multiply both errors and costs.
| Challenge | Impact |
|---|---|
| Compound Error Propagation | 80% accuracy × 80% × 80% = 51.2% overall accuracy for 3-agent chain |
| Context Degradation | Critical information lost or misinterpreted at each hop |
| Lossy NL Interface | Natural language between agents adds semantic drift and ambiguity |
| Latency Multiplication | Each layer adds network + inference time (3× latency for 3 levels) |
| Operational Complexity | Managing multiple agents far more complex than single agent |
| State Management Overhead | Maintaining context across agents requires sophisticated orchestration |
This class of solutions focuses on limiting the number of tools available at any moment, rather than helping the LLM distinguish between them.
Core Principle: If tool selection accuracy decreases with more tools, reducing the tool count improves accuracy.
These solutions typically achieve tool counts in the optimal range (under 10-15 tools) where LLMs demonstrate the highest selection accuracy.
Concept: Combine multiple low-level operations into higher-level business domain tools.
Before (Granular Tools):
database.query_orderdatabase.query_itemspayment.validate_paymentinventory.check_stockemail.send_confirmationdatabase.update_order_status
After (Abstracted Tool):
order.create_order(internally executes all the above steps in sequence)
This shifts complexity from LLM decision-making into deterministic code execution within the tool implementation.
Each abstracted tool encapsulates a complete business workflow:
- High-level parameters (
customer_id,product_list) - Predefined logic, error handling, and sequencing
- Internal orchestration of all low-level operations
The LLM makes one decision instead of multiple sequential tool selections.
-
Reduced Decision Complexity: Instead of making multiple sequential tool selection decisions (which tool to call first, second, third, etc.), the LLM makes a single decision to invoke one abstracted tool. This dramatically reduces the cognitive load and opportunity for errors in sequencing.
-
Search Space Reduction: With abstraction, the LLM's tool selection search space shrinks significantly. Choosing between 5-10 high-level business operations is substantially easier than choosing between 50-100 low-level component operations, leading to higher accuracy.
-
Eliminates Sequencing Errors: LLMs can struggle with correctly ordering multi-step operations, especially when dealing with dependencies or conditional logic. Abstracted tools eliminate this problem by encoding the correct sequence in deterministic code that always executes in the proper order.
-
Clearer Semantic Boundaries: Business-level abstractions naturally align with how users express their needs in natural language. When a user says "create an order," a single
create_ordertool is semantically clearer than requiring the LLM to decompose this into multiple database queries, validation checks, and notifications. -
Reduced Token Overhead: Fewer tools mean less token usage in the system prompt/API describing available tools, leaving more context window for the actual conversation and user intent processing.
-
Better Error Recovery: When errors occur within an abstracted tool, they can be handled internally with proper rollback logic and compensating transactions, rather than leaving the LLM to handle partial failures across multiple tool calls.
-
Consistent Execution: The deterministic nature of abstracted tools ensures that the same input always produces the same sequence of operations, eliminating variability introduced by LLM non-determinism in multi-step workflows.
| Aspect | Best Practice |
|---|---|
| Transaction Management | Implement proper atomicity (ACID) or eventual consistency (BASE) |
| Error Handling | Robust internal handling with clear error messages for the LLM |
| Parameter Design | Capture business intent, not technical details (e.g., create_order(customer_id, items)) |
| Granularity Balance | Abstract at business capability level (not too coarse, not too fine) |
| Idempotency | Make tools safe for retry if LLM calls multiple times |
- Reduces tool count by an order of magnitude
- Improves deterministic behavior (well-defined execution order)
- Simplifies prompt engineering (focus on business outcomes)
- Reduces total latency (eliminates multiple LLM round-trips)
- Easier testing (business logic in code, not emergent from LLM)
- May lose flexibility for edge cases or creative tool combinations
- Requires careful workflow analysis for abstraction boundaries
- Business process changes require tool implementation updates
- Risk of tool proliferation if every workflow variation needs separate tools
Concept: Dynamically provide only relevant tools based on context, using intelligent filtering and embedding-based routing.
Real-world implementations: GitHub Copilot | Claude's Tool Search
Rather than exposing all tools simultaneously, this approach strategically groups similar tools and uses intelligent filtering to present only relevant subsets.
Similar tools are functionally grouped under "virtual tools" that act like directories containing related tools. The LLM sees these virtual tools first and can expand (more on this later) them only when needed.
As an example GitHub Copilot's implementation uses embedding-based clustering:
- Generate embeddings for each tool using a specialized embedding model (like GitHub's Copilot embedding model)
- Group tools using cosine similarity in the embedding space
- Use LLM to summarize each cluster (much faster than asking LLM to categorize from scratch)
- Cache tool embeddings and group summaries locally for fast recomputation
Example grouping:
Virtual Tool: "Order Management Tools"
├── create_order
├── update_order
├── cancel_order
├── get_order_status
└── list_orders
Virtual Tool: "Order Fulfillment Tools"
├── ship_order
├── track_shipment
└── update_delivery_status
Context-aware pre-selection eliminates unnecessary exploratory calls. Before any virtual tool is expanded, the system:
- Compares the query embedding against vector representations of all tools and their clusters
- Pre-selects the most semantically relevant candidates
- Includes likely-needed tools directly in the candidate set
Example scenario:
- User query: "Ship the order and send tracking notification to customer"
- Without routing: Model expands on search tools → customer tools → inventory tools → finally order fulfillment tools (multiple round trips)
- With routing: System infers from query embedding that ship_order tool is needed and includes it directly (single round trip)
Based on usage statistics and performance data, identify a minimal core toolset of essential, frequently-used tools. Group remaining specialized tools into virtual categories that expand on-demand.
Example implementation approach:
- Core toolset: 10-15 essential tools (search, filter, basic CRUD operations for primary entities)
- Virtual groups: Order Management Tools, Order Fulfillment Tools, Payment Tools, Inventory Tools
- Tools expanded only when semantically relevant to the task
Grouping similar tools together reduces cache misses since related tools are typically used together in workflows. When one tool in a group is needed, others are likely relevant too.
Handling Cache Misses:
Despite 94.5% accuracy (GitHub's results), the system must handle cases where wrong tools are initially loaded:
- Progressive Expansion: Auto-expand relevant groups when model attempts unavailable tools
- Exploratory Tool Calls: Model can invoke virtual tools to see their contents (
expand_order_tools()) - Search and Discovery: Provide
search_tools(query)to find tools across all groups - Semantic Error Messages: Suggest similar tools from other groups when tool not found
- Learning from Misses: Log and analyze misses to improve routing
Example cache miss flow:
User: "Process a refund for order #12345"
→ System pre-expands: Core + Order Management Tools (incorrect)
→ Model attempts: tool for a refund
→ System detects: Tool not visible, belongs to "Payment Tools"
→ System auto-expands: Payment Tools group
→ Model successfully calls: process_refund()
Cost: One extra round-trip vs. massive load of all tools upfront
Benchmarks (GitHub) show dramatic improvements in Tool Use Coverage:
| Approach | Coverage | Tool Pre-Expansion Success |
|---|---|---|
| Embedding-based | 94.5% | 72% of tool calls |
| LLM-based selection | 87.5% | - |
| Default static list | 69.0% | 19% of tool calls |
Result: 3.8× improvement in pre-expanded tool calls with embedding-based matching
| Metric | Improvement |
|---|---|
| Time To First Token (TTFT) | -190ms |
| Time To Final Token | -400ms |
| Benchmark Resolution Rates | +2-5 percentage points |
-
Cognitive Load Management: Models perform better with 10-15 tools than with 40+ tools. Selective filtering keeps the working set small.
-
Reduced Decision Fatigue: Fewer options mean clearer reasoning paths and less opportunity for selection errors or tool misuse.
-
Context Window Efficiency: Tool definitions consume valuable context. Filtering saves tokens for actual task-relevant information.
-
Improved Accuracy: Testing shows agents with oversized toolsets ignore instructions, use tools incorrectly, and call unnecessary tools. Smaller, curated sets prevent these issues.
-
Semantic Relevance: Embedding-based routing ensures tools match user intent more precisely than exposing everything and hoping the model chooses correctly.
| Aspect | Consideration |
|---|---|
| Embedding Model | Requires high-quality model optimized for semantic similarity |
| Grouping Strategy | Balance between too few groups (excessive expansion) and too many (defeats purpose) |
| Caching | Cache embeddings and group summaries locally for fast recomputation |
| Fallback Mechanisms | Graceful handling when filtering is too aggressive; allow tool requests |
- Improvement in tool use coverage
- Faster average response times
- Improvement in success rates
- Handles hundreds of tools without performance degradation
- Eliminates "optimizing tool selection" delays
- Lower token costs (fewer tool definitions in prompts)
- Needs embedding generation and similarity search infrastructure
- Requires access to good embedding models
- Finding optimal cluster sizes requires experimentation
- Cold start problem (initial queries lack context)
- Tool groups need updating as new tools are added
✅ Best suited for:
- Systems with dozens or hundreds of tools
- Identifiable tool usage patterns
- Agents serving diverse use cases
- Performance-sensitive applications
- Production systems with embedding infrastructure
❌ May not be necessary for:
- Small tool sets (< 15 tools)
- Single-purpose agents with fixed requirements
- Prototyping before usage patterns emerge
- Resource-constrained environments without embedding infrastructure
Concept: Convert MCP tools into programming language APIs and have LLMs write code to call them instead of using special tool-calling tokens.
Read more: Cloudflare's Code Mode | Anthropic's Code Execution with MCP
This "code mode" approach leverages LLMs' strength in writing code rather than their relatively weaker tool-calling capabilities.
Traditional Tool Calling:
<|tool_call|>
{"name": "order.getOrderDetails", "arguments": {"orderId": "12345"}}
<|end_tool_call|>
Code-Based Approach:
from servers.order_management import get_order_details
order_data = get_order_details({'orderId': '12345'})The LLM discovers tools through filesystem navigation or search, loads only needed definitions, and writes code executed in a secure sandbox.
Load only needed tools (typically 2,000 tokens) instead of all tools upfront (150,000+ tokens in extreme cases).
Token Reduction: 98.7% (for large context tools)
With traditional tool calling, a 50,000-token document flowing through two tool calls consumes 100,000+ tokens. With code execution, intermediate data stays in the execution environment only final results return to the model:
# Data flows between tools without entering model context
order_data = order.get_order_details({'orderId': '12345'})
order.update_status({'orderId': '12345', 'status': 'shipped'})
print('Order updated successfully') # Only this enters model contextStandard programming patterns execute without LLM involvement:
# Filter 10,000 rows in execution environment
all_orders = order.get_all_orders()
pending = [order for order in all_orders if order["Status"] == 'pending']
print(f'Found {len(pending)} pending orders') # Model only sees summary| Feature | Benefit |
|---|---|
| Binding-based access | Sandbox has no network access, only specific MCP server bindings |
| Automatic PII tokenization | Sensitive data tokenized before reaching model |
| No API key exposure | Keys managed by execution harness, never visible to LLM |
Agents can save intermediate results to files and build libraries of reusable functions:
# Save once as a skill
def export_orders_as_csv(status: str) -> str:
orders = order.get_orders_by_status({'status': status})
# ... convert and save ...
return file_path
# Reuse across sessions
from skills.export_orders_as_csv import export_orders_as_csv| Reason | Explanation |
|---|---|
| Training Data Advantage | Millions of lines of real code vs. synthetic tool-calling examples |
| Reduced Cognitive Load | order.get_order_details() more natural than special tokens with JSON |
| Scale Handling | Supports hundreds of tools through progressive discovery |
| Multi-Step Efficiency | Eliminates LLM processing for each intermediate step |
- 98%+ token reduction through progressive discovery and in-environment processing
- Lower latency (eliminates multiple LLM inference round-trips)
- Better handling of large datasets (transformation happens outside context window)
- Stronger security (PII tokenization, binding-based access control)
- Reusable skill libraries that improve over time
- Requires secure sandbox infrastructure (isolates, wasm, containers, or VMs)
- More complex to implement and operate than direct tool calling
- Debugging generated code can be more difficult
- Need resource limits for long-running or memory-intensive code
✅ Code-based approach excels for:
- Multiple sequential tool calls
- Large datasets exceeding context windows
- Complex conditional logic or loops
- Agents with dozens/hundreds of tools
- Privacy-sensitive operations
Traditional tool calling suffices for:
- Simple single-tool operations
- Small tool sets (<10)
- Small intermediate results
- Prototyping phases
Combining these strategies creates a comprehensive system for maximizing tool selection accuracy:
- Make tools discoverable → Clear naming, documentation, and examples
- Reduce cognitive load → Filtering, grouping, and abstraction
- Learn from usage → Analytics, feedback, and continuous improvement
- Adapt to context → Understanding user intent
- Guide intelligently → Validation, suggestions, and relationship awareness
Key Takeaway: No single strategy solves all problems. A thoughtful combination tailored to your use case will significantly improve both tool selection accuracy and overall agent performance.
- modelcontextprotocol/modelcontextprotocol#1300
- https://github.blog/ai-and-ml/github-copilot/how-were-making-github-copilot-smarter-with-fewer-tools/
- https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool
- https://blog.cloudflare.com/code-mode/
- https://www.anthropic.com/engineering/code-execution-with-mcp