Skip to content

Instantly share code, notes, and snippets.

@suneetnangia
Last active January 13, 2026 13:44
Show Gist options
  • Select an option

  • Save suneetnangia/f5db84420e268717d074d6037efd7311 to your computer and use it in GitHub Desktop.

Select an option

Save suneetnangia/f5db84420e268717d074d6037efd7311 to your computer and use it in GitHub Desktop.
Tools Accuracy and LLMs

S/LLMs and Tools Accuracy

As AI agents gain access to more tools and capabilities, a critical challenge emerges: the more tools an LLM can choose from, the less accurate it becomes at selecting the right one. This inverse relationship between tool quantity and selection accuracy fundamentally limits how many capabilities we can expose to a single agent.

This guide examines proven strategies to overcome this limitation, drawing from publicly available research and real-world implementations from GitHub, Anthropic, and Cloudflare. We explore both techniques that help LLMs better understand available tools and approaches that strategically reduce the number of tools presented at any given time.


Table of Contents


The Challenge

Three fundamental constraints shape how we expose tools to LLMs:

Constraint Description Impact
Platform Limitations Most LLMs support max 128 tools through structured API calling Hard ceiling on capabilities
Accuracy Degradation Tool selection accuracy drops progressively beyond ~10 tools Practical ceiling far below platform limits
Resource Overhead Each tool consumes tokens in system prompts Higher costs, reduced context for tasks

Key Insight: The tension isn't just about how many tools to expose, but achieving the right balance between capability breadth and selection accuracy.


Solution Approaches Overview

Two main categories of solutions address the tool selection accuracy challenge:

🎯 Semantic Solutions

Help the LLM better understand and distinguish between tools based on their semantic meaning.

  • Namespaces: Organize tools using hierarchical naming
  • Additional Strategies: Enhanced documentation, few-shot learning, tool ranking, and more
  • Hierarchical Agents: Distribute tools across specialized agents

📉 Tools Reduction Solutions

Limit the actual number of tools the LLM needs to choose from at any given moment.

  • Abstraction: Combine multiple operations into higher-level business tools
  • Grouping and Filtering: Dynamically present only relevant tool subsets

Semantic Solutions

Namespaces

Concept: Organize tools using hierarchical naming conventions with dot notation or path-like structures.

Example Structure (Tool Names)

  • order.management.create_order
  • order.management.delete_order
  • order.fulfillment.ship_order
  • order.fulfillment.track_shipment
  • order.payment.process_payment
  • order.payment.refund_payment

Shown in hierarchical view as below:

order/
├── management/
│   ├── create_order
│   └── delete_order
├── fulfillment/
│   ├── ship_order
│   └── track_shipment
└── payment/
    ├── process_payment
    └── refund_payment

How It Works

LLMs are designed to predict the next token. Namespacing aligns perfectly with this capability. When processing order.payment.process_payment:

  1. First token: order → establishes the domain
  2. Second token: payment → narrows to payment operations
  3. Third token: process_payment → identifies the specific action

This hierarchical structure provides clear semantic grouping and helps the LLM understand relationships between tools.

Why LLMs Work Better with Namespaces

  1. Attention Mechanism Alignment: LLMs use attention mechanisms to identify patterns and relationships. Namespaced tools create clear structural patterns (domain.subdomain.action) that the attention layers can easily recognize and differentiate, reducing confusion between similarly-named tools.

  2. Semantic Clustering in Embedding Space: During training, LLMs learn that words appearing in similar contexts have related meanings. Namespaces explicitly group related tools together (e.g., all order.payment.* tools), which helps the model's internal representations cluster semantically related operations, making the right tool more likely to have higher activation.

  3. Reduced Ambiguity: Without namespaces, tools like create_order, create_customer, create_product all start with "create" and look similar to the model. With namespaces like order.management.create_order, customer.management.create_customer, the domain context is embedded in the tool name itself, drastically reducing ambiguity.

  4. Prefix-Based Pruning: When the LLM begins generating a tool name, the namespace prefix acts as a constraint. Once it generates order.payment., it has effectively pruned all tools outside that namespace from consideration, narrowing the search space dramatically and improving selection accuracy.

  5. Training Data Similarity: Most codebases and APIs in the training data use similar hierarchical conventions (package.module.function, namespace.class.method), so LLMs have seen countless examples of this pattern and learned to process it effectively.

  6. Compositional Understanding: LLMs excel at compositional semantics - understanding meaning built from parts. A namespaced tool name like inventory.warehouse.check_stock is compositional: the model understands "inventory" + "warehouse" + "check stock" as a natural combination, whereas isolated tool names require the model to infer context from descriptions alone.

✅ Benefits

Benefit Description
Improved Context Namespace prefix provides immediate domain context
Better Differentiation Similar tools in different domains easily distinguished
Scalability Organizational structure remains clear as tools grow
Single Agent Works within one LLM call without latency/cost overhead
Natural Token Prediction Aligns with LLM's core sequential processing strength

⚠️ Considerations

  • Requires consistent naming conventions across all tools
  • Tool names become longer, slightly increasing token usage
  • Works best when combined with good descriptions that reinforce the namespace hierarchy

Additional Strategies

Beyond namespacing, several complementary techniques enhance tool selection accuracy through better understanding:

1. Enhanced Tool Documentation

Provide rich contextual information with structured metadata:

{
  "name": "process_order_shipment",
  "when_to_use": "When an order is ready to be shipped to the customer",
  "when_not_to_use": "Don't use for order creation or already shipped orders",
  "categories": ["order_management", "fulfillment", "logistics"],
  "prerequisites": ["order in 'confirmed' status", "payment completed"],
  "complexity": "medium"
}

2. Few-Shot Learning

Include 2-3 examples of successful tool usage in the system prompt:

Example: User wants to "ship an order and notify customer"
→ Use: get_order_details(order_id) → process_shipment(order_id) → send_notification(customer_id)

3. Usage-Based Tool Ranking

Why tool order matters: LLMs favor tools presented first (primacy effect) and last (recency effect). Tools in early positions are selected 2-3× more often than those beyond position 10.

Impact: Tool selection accuracy varies by 10-15% based on ordering alone. Frequency-based ordering improves success rates by 5-12%.

Best practices:

  • Place core/frequently-used tools first
  • Dynamically reorder based on query context
  • Prioritize tools with higher success rates
  • Boost recently-used tools within current session

Example:

Query: "Complete order shipment"
Reordered: [ship_order, track_shipment, send_notification, ...]
Not: [create_order, cancel_order, ship_order, ...]

4. User Preferences and Personalization

  • Track individual user's tool preferences and usage patterns
  • Adapt based on user's role (developer, designer, manager)
  • Allow users to mark favorite tools or hide irrelevant ones

5. Feedback Loops and Continuous Improvement

  • Monitor tool selection accuracy in production
  • Track failure rates and error patterns
  • A/B test different descriptions, groupings, or orderings
  • Fine-tune models on domain-specific tool usage patterns

6. Semantic Tool Relationships

Define explicit relationships between tools:

Tool: create_order
Related tools:
  - Next steps: process_payment, update_order_status
  - Requires: validate_customer
  - Conflicts with: cancel_order (on same order)

7. Natural Language Understanding

  • Intent Classification: Pre-classify queries into categories, activate only relevant tool categories
  • Entity Extraction: Extract key entities, pre-filter tools compatible with detected entities

Hierarchical Agents

Concept: Create a hierarchy of specialized agents where each handles a specific domain, similar to Domain-Driven Design (DDD) in microservices.

Note: This approach combines semantic routing with tools reduction through abstraction—each agent only sees relevant tools for their domain.

Instead of a single monolithic agent, the system distributes responsibility across multiple specialized agents. Requests are progressively routed from general orchestration agents down to specialized domain agents that possess the specific tools needed.

Example Architecture

Orchestration Agent
└── Ordering Agent
    ├── Order Management Agent
    │   ├── create_order
    │   └── delete_order
    ├── Order Fulfillment Agent
    │   ├── ship_order
    │   └── track_shipment
    └── Order Payment Agent
        ├── process_payment
        └── refund_payment

How It Works

The goal is to prevent any single agent from being overloaded with tools. Requests flow from top to bottom:

  1. Orchestration Agent: Routes user request to appropriate domain agent
  2. Domain Agent: Routes to appropriate sub-domain agent
  3. Specialist Agent: Executes the actual tool call

Why Hierarchical Agents Can Be Effective

  1. Cognitive Load Reduction: By distributing tools across multiple specialized agents, each individual LLM operates with a smaller, more focused tool set. This mirrors human cognitive limits - specialists perform better in narrow domains than generalists trying to handle everything.

  2. Context Specialization: Each agent can maintain domain-specific context, system prompts, and examples optimized for its specialized area. An "Order Payment Agent" can have payment-specific instructions, validation rules, and error handling that would clutter a general-purpose agent.

  3. Parallel Processing Potential: Multiple leaf agents can theoretically process independent requests in parallel, improving throughput in high-load scenarios (though this adds complexity).

  4. Tool Set Isolation: Changes to tools in one domain don't affect other domains. Adding new payment tools doesn't increase the cognitive load on the fulfillment or management agents.

  5. Explicit Reasoning Chain: The routing decisions create an audit trail showing how the system arrived at a particular tool selection, which aids debugging and system improvement.

  6. Domain Expertise Modeling: Different agents can use different base models, prompts, or even fine-tuned versions optimized for their specific domain, potentially improving accuracy within each specialization.

⚠️ Technical Challenges

Critical Consideration: Multiple LLMs in a chain multiply both errors and costs.

Challenge Impact
Compound Error Propagation 80% accuracy × 80% × 80% = 51.2% overall accuracy for 3-agent chain
Context Degradation Critical information lost or misinterpreted at each hop
Lossy NL Interface Natural language between agents adds semantic drift and ambiguity
Latency Multiplication Each layer adds network + inference time (3× latency for 3 levels)
Operational Complexity Managing multiple agents far more complex than single agent
State Management Overhead Maintaining context across agents requires sophisticated orchestration

Tools Reduction Solutions

This class of solutions focuses on limiting the number of tools available at any moment, rather than helping the LLM distinguish between them.

Core Principle: If tool selection accuracy decreases with more tools, reducing the tool count improves accuracy.

These solutions typically achieve tool counts in the optimal range (under 10-15 tools) where LLMs demonstrate the highest selection accuracy.


Abstraction

Concept: Combine multiple low-level operations into higher-level business domain tools.

Example Transformation

Before (Granular Tools):

  • database.query_order
  • database.query_items
  • payment.validate_payment
  • inventory.check_stock
  • email.send_confirmation
  • database.update_order_status

After (Abstracted Tool):

  • order.create_order (internally executes all the above steps in sequence)

This shifts complexity from LLM decision-making into deterministic code execution within the tool implementation.

How It Works

Each abstracted tool encapsulates a complete business workflow:

  • High-level parameters (customer_id, product_list)
  • Predefined logic, error handling, and sequencing
  • Internal orchestration of all low-level operations

The LLM makes one decision instead of multiple sequential tool selections.

Why LLMs Benefit from Abstraction

  1. Reduced Decision Complexity: Instead of making multiple sequential tool selection decisions (which tool to call first, second, third, etc.), the LLM makes a single decision to invoke one abstracted tool. This dramatically reduces the cognitive load and opportunity for errors in sequencing.

  2. Search Space Reduction: With abstraction, the LLM's tool selection search space shrinks significantly. Choosing between 5-10 high-level business operations is substantially easier than choosing between 50-100 low-level component operations, leading to higher accuracy.

  3. Eliminates Sequencing Errors: LLMs can struggle with correctly ordering multi-step operations, especially when dealing with dependencies or conditional logic. Abstracted tools eliminate this problem by encoding the correct sequence in deterministic code that always executes in the proper order.

  4. Clearer Semantic Boundaries: Business-level abstractions naturally align with how users express their needs in natural language. When a user says "create an order," a single create_order tool is semantically clearer than requiring the LLM to decompose this into multiple database queries, validation checks, and notifications.

  5. Reduced Token Overhead: Fewer tools mean less token usage in the system prompt/API describing available tools, leaving more context window for the actual conversation and user intent processing.

  6. Better Error Recovery: When errors occur within an abstracted tool, they can be handled internally with proper rollback logic and compensating transactions, rather than leaving the LLM to handle partial failures across multiple tool calls.

  7. Consistent Execution: The deterministic nature of abstracted tools ensures that the same input always produces the same sequence of operations, eliminating variability introduced by LLM non-determinism in multi-step workflows.

Implementation Considerations

Aspect Best Practice
Transaction Management Implement proper atomicity (ACID) or eventual consistency (BASE)
Error Handling Robust internal handling with clear error messages for the LLM
Parameter Design Capture business intent, not technical details (e.g., create_order(customer_id, items))
Granularity Balance Abstract at business capability level (not too coarse, not too fine)
Idempotency Make tools safe for retry if LLM calls multiple times

✅ Benefits

  • Reduces tool count by an order of magnitude
  • Improves deterministic behavior (well-defined execution order)
  • Simplifies prompt engineering (focus on business outcomes)
  • Reduces total latency (eliminates multiple LLM round-trips)
  • Easier testing (business logic in code, not emergent from LLM)

⚠️ Considerations

  • May lose flexibility for edge cases or creative tool combinations
  • Requires careful workflow analysis for abstraction boundaries
  • Business process changes require tool implementation updates
  • Risk of tool proliferation if every workflow variation needs separate tools

Grouping and Filtering

Concept: Dynamically provide only relevant tools based on context, using intelligent filtering and embedding-based routing.

Real-world implementations: GitHub Copilot | Claude's Tool Search

Rather than exposing all tools simultaneously, this approach strategically groups similar tools and uses intelligent filtering to present only relevant subsets.

Key Implementation Strategies

1. Virtual Tools and Adaptive Clustering

Similar tools are functionally grouped under "virtual tools" that act like directories containing related tools. The LLM sees these virtual tools first and can expand (more on this later) them only when needed.

As an example GitHub Copilot's implementation uses embedding-based clustering:

  • Generate embeddings for each tool using a specialized embedding model (like GitHub's Copilot embedding model)
  • Group tools using cosine similarity in the embedding space
  • Use LLM to summarize each cluster (much faster than asking LLM to categorize from scratch)
  • Cache tool embeddings and group summaries locally for fast recomputation

Example grouping:

Virtual Tool: "Order Management Tools"
  ├── create_order
  ├── update_order
  ├── cancel_order
  ├── get_order_status
  └── list_orders

Virtual Tool: "Order Fulfillment Tools"
  ├── ship_order
  ├── track_shipment
  └── update_delivery_status
2. Embedding-Guided Tool Routing

Context-aware pre-selection eliminates unnecessary exploratory calls. Before any virtual tool is expanded, the system:

  1. Compares the query embedding against vector representations of all tools and their clusters
  2. Pre-selects the most semantically relevant candidates
  3. Includes likely-needed tools directly in the candidate set

Example scenario:

  • User query: "Ship the order and send tracking notification to customer"
  • Without routing: Model expands on search tools → customer tools → inventory tools → finally order fulfillment tools (multiple round trips)
  • With routing: System infers from query embedding that ship_order tool is needed and includes it directly (single round trip)
3. Core Toolset with Expandable Groups

Based on usage statistics and performance data, identify a minimal core toolset of essential, frequently-used tools. Group remaining specialized tools into virtual categories that expand on-demand.

Example implementation approach:

  • Core toolset: 10-15 essential tools (search, filter, basic CRUD operations for primary entities)
  • Virtual groups: Order Management Tools, Order Fulfillment Tools, Payment Tools, Inventory Tools
  • Tools expanded only when semantically relevant to the task

Technical Advantages

1. Reduced Cache Miss Rate

Grouping similar tools together reduces cache misses since related tools are typically used together in workflows. When one tool in a group is needed, others are likely relevant too.

Handling Cache Misses:

Despite 94.5% accuracy (GitHub's results), the system must handle cases where wrong tools are initially loaded:

  1. Progressive Expansion: Auto-expand relevant groups when model attempts unavailable tools
  2. Exploratory Tool Calls: Model can invoke virtual tools to see their contents (expand_order_tools())
  3. Search and Discovery: Provide search_tools(query) to find tools across all groups
  4. Semantic Error Messages: Suggest similar tools from other groups when tool not found
  5. Learning from Misses: Log and analyze misses to improve routing

Example cache miss flow:

User: "Process a refund for order #12345"
→ System pre-expands: Core + Order Management Tools (incorrect)
→ Model attempts: tool for a refund
→ System detects: Tool not visible, belongs to "Payment Tools"
→ System auto-expands: Payment Tools group
→ Model successfully calls: process_refund()
Cost: One extra round-trip vs. massive load of all tools upfront
2. Embedding-Based Selection Performance

Benchmarks (GitHub) show dramatic improvements in Tool Use Coverage:

Approach Coverage Tool Pre-Expansion Success
Embedding-based 94.5% 72% of tool calls
LLM-based selection 87.5% -
Default static list 69.0% 19% of tool calls

Result: 3.8× improvement in pre-expanded tool calls with embedding-based matching

3. Significant Latency Reduction (GitHub)
Metric Improvement
Time To First Token (TTFT) -190ms
Time To Final Token -400ms
Benchmark Resolution Rates +2-5 percentage points

Why LLMs Benefit from Grouping and Filtering

  1. Cognitive Load Management: Models perform better with 10-15 tools than with 40+ tools. Selective filtering keeps the working set small.

  2. Reduced Decision Fatigue: Fewer options mean clearer reasoning paths and less opportunity for selection errors or tool misuse.

  3. Context Window Efficiency: Tool definitions consume valuable context. Filtering saves tokens for actual task-relevant information.

  4. Improved Accuracy: Testing shows agents with oversized toolsets ignore instructions, use tools incorrectly, and call unnecessary tools. Smaller, curated sets prevent these issues.

  5. Semantic Relevance: Embedding-based routing ensures tools match user intent more precisely than exposing everything and hoping the model chooses correctly.

Implementation Considerations

Aspect Consideration
Embedding Model Requires high-quality model optimized for semantic similarity
Grouping Strategy Balance between too few groups (excessive expansion) and too many (defeats purpose)
Caching Cache embeddings and group summaries locally for fast recomputation
Fallback Mechanisms Graceful handling when filtering is too aggressive; allow tool requests

✅ Benefits

  • Improvement in tool use coverage
  • Faster average response times
  • Improvement in success rates
  • Handles hundreds of tools without performance degradation
  • Eliminates "optimizing tool selection" delays
  • Lower token costs (fewer tool definitions in prompts)

⚠️ Challenges

  • Needs embedding generation and similarity search infrastructure
  • Requires access to good embedding models
  • Finding optimal cluster sizes requires experimentation
  • Cold start problem (initial queries lack context)
  • Tool groups need updating as new tools are added

When to Use

✅ Best suited for:

  • Systems with dozens or hundreds of tools
  • Identifiable tool usage patterns
  • Agents serving diverse use cases
  • Performance-sensitive applications
  • Production systems with embedding infrastructure

❌ May not be necessary for:

  • Small tool sets (< 15 tools)
  • Single-purpose agents with fixed requirements
  • Prototyping before usage patterns emerge
  • Resource-constrained environments without embedding infrastructure

API Calling vs Tool Calling Approach

Concept: Convert MCP tools into programming language APIs and have LLMs write code to call them instead of using special tool-calling tokens.

Read more: Cloudflare's Code Mode | Anthropic's Code Execution with MCP

This "code mode" approach leverages LLMs' strength in writing code rather than their relatively weaker tool-calling capabilities.

Traditional vs Code-Based Approach

Traditional Tool Calling:

<|tool_call|>
{"name": "order.getOrderDetails", "arguments": {"orderId": "12345"}}
<|end_tool_call|>

Code-Based Approach:

from servers.order_management import get_order_details
order_data = get_order_details({'orderId': '12345'})

The LLM discovers tools through filesystem navigation or search, loads only needed definitions, and writes code executed in a secure sandbox.

Key Technical Advantages

1. Progressive Tool Discovery

Load only needed tools (typically 2,000 tokens) instead of all tools upfront (150,000+ tokens in extreme cases).

Token Reduction: 98.7% (for large context tools)

2. Context Window Efficiency

With traditional tool calling, a 50,000-token document flowing through two tool calls consumes 100,000+ tokens. With code execution, intermediate data stays in the execution environment only final results return to the model:

# Data flows between tools without entering model context
order_data = order.get_order_details({'orderId': '12345'})
order.update_status({'orderId': '12345', 'status': 'shipped'})
print('Order updated successfully')  # Only this enters model context

3. Advanced Control Flow

Standard programming patterns execute without LLM involvement:

# Filter 10,000 rows in execution environment
all_orders = order.get_all_orders()
pending = [order for order in all_orders if order["Status"] == 'pending']
print(f'Found {len(pending)} pending orders')  # Model only sees summary

4. Security and Privacy

Feature Benefit
Binding-based access Sandbox has no network access, only specific MCP server bindings
Automatic PII tokenization Sensitive data tokenized before reaching model
No API key exposure Keys managed by execution harness, never visible to LLM

5. State Persistence and Reusable Skills

Agents can save intermediate results to files and build libraries of reusable functions:

# Save once as a skill
def export_orders_as_csv(status: str) -> str:
    orders = order.get_orders_by_status({'status': status})
    # ... convert and save ...
    return file_path

# Reuse across sessions
from skills.export_orders_as_csv import export_orders_as_csv

Why LLMs Perform Better with Code

Reason Explanation
Training Data Advantage Millions of lines of real code vs. synthetic tool-calling examples
Reduced Cognitive Load order.get_order_details() more natural than special tokens with JSON
Scale Handling Supports hundreds of tools through progressive discovery
Multi-Step Efficiency Eliminates LLM processing for each intermediate step

Benefits and Trade-offs

✅ Benefits

  • 98%+ token reduction through progressive discovery and in-environment processing
  • Lower latency (eliminates multiple LLM inference round-trips)
  • Better handling of large datasets (transformation happens outside context window)
  • Stronger security (PII tokenization, binding-based access control)
  • Reusable skill libraries that improve over time

⚠️ Challenges

  • Requires secure sandbox infrastructure (isolates, wasm, containers, or VMs)
  • More complex to implement and operate than direct tool calling
  • Debugging generated code can be more difficult
  • Need resource limits for long-running or memory-intensive code

When to Use

✅ Code-based approach excels for:

  • Multiple sequential tool calls
  • Large datasets exceeding context windows
  • Complex conditional logic or loops
  • Agents with dozens/hundreds of tools
  • Privacy-sensitive operations

Traditional tool calling suffices for:

  • Simple single-tool operations
  • Small tool sets (<10)
  • Small intermediate results
  • Prototyping phases

Summary

Combining these strategies creates a comprehensive system for maximizing tool selection accuracy:

  1. Make tools discoverable → Clear naming, documentation, and examples
  2. Reduce cognitive load → Filtering, grouping, and abstraction
  3. Learn from usage → Analytics, feedback, and continuous improvement
  4. Adapt to context → Understanding user intent
  5. Guide intelligently → Validation, suggestions, and relationship awareness

Key Takeaway: No single strategy solves all problems. A thoughtful combination tailored to your use case will significantly improve both tool selection accuracy and overall agent performance.


References and Further Reads

  1. modelcontextprotocol/modelcontextprotocol#1300
  2. https://github.blog/ai-and-ml/github-copilot/how-were-making-github-copilot-smarter-with-fewer-tools/
  3. https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool
  4. https://blog.cloudflare.com/code-mode/
  5. https://www.anthropic.com/engineering/code-execution-with-mcp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment